2024.9.2.0

Instagram: add options to enable/disable image extraction from video
OnlyFans: update to the changed API
YouTube: videos are parsed from the 'featured', not from the 'videos' page
Feed: add prompt before moving entire feed/session
MainFrame: add 'Alt+U' and 'Ctrl+U' to open the user search form
UserImage: user image creation update

CONTRIBUTING.md

@@ -1,17 +1,14 @@
 # Contributor's Guide
 
-I welcome requests! Follow these steps to contribute:
+Follow these steps to contribute:
-
 1. Find an [issue](https://github.com/AAndyProgram/SCrawler/issues) that needs assistance.
-1. Let me know you are working on it by posting a comment on the issue.
+1. Let me know you're working on this by posting a comment on this issue.
-1. If you find an error in the code, please provide a link to the file and the line number.
+1. If you find a bug in the code, please provide a link to the file and line number.
 1. If you have a code change suggestion, you can post a replacement code block.<!-- I also accept pull requests.-->
 
 # How to report a problem
-1. Attach the **profile URLs or links** that you cannot download.
+
-1. Attach the **LOG** if it exists.
+**[Read here](https://github.com/AAndyProgram/SCrawler/blob/main/FAQ.md#how-to-report-a-problem)**
-1. **Attach the environment information copied from SCrawler (click the top right info button in the main window, then the `Environment` button, then the `Copy` button, and paste the copied text into the issue).**
-1. *Add screenshots to illustrate the problem (**optional**)*
 
 # How to build from source
 1. Delete the `PersonalUtilities` project from the solution.
@@ -30,7 +27,7 @@ I welcome requests! Follow these steps to contribute:
 **I'm currently not accepting requests to develop new sites.**
 
 1. Check [issues](https://github.com/AAndyProgram/SCrawler/issues) (open and [closed](https://github.com/AAndyProgram/SCrawler/issues?q=is%3Aissue+is%3Aclosed)) and [discussions](https://github.com/AAndyProgram/SCrawler/discussions) to find your issue. Perhaps I have already answered your request.
-1. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
+1. If you don't find anything, create a new issue with your request.
 
 # Requirements for new site requests
 

Changelog.md

@@ -1,3 +1,31 @@
+# 2024.9.2.0
+
+*2024-09-02*
+
+- Added
+  - Instagram: options to enable/disable image extraction from video
+  - Feed: **prompt before moving entire feed/session**
+  - Main window: hotkeys `Alt+U` and `Ctrl+U` to open the user search form
+  - Minor improvements
+- Updated
+  - gallery-dl up to version **1.27.3**
+- Fixed
+  - **OnlyFans**: data is not downloading
+  - YouTube (SCrawler): incorrect parsing of video page
+  - Minor bugs
+
+# 2024.8.10.0
+
+*2024-08-10*
+
+- Added
+  - Feed: button to open file folder
+- Updated
+  - yt-dlp up to version **2024.08.06**
+  - gallery-dl up to version **1.27.2**
+- Fixed
+  - YouTube (standalone app): **video is being parsed using cookies but is not downloading** *(Issue #205)*
+
 # 2024.8.1.0
 
 *2024-08-01*

FAQ.md

@@ -1,120 +1,98 @@
-# Frequently asked questions
 
-**Please read the [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) Before asking a question!**
+**Join our Discord server**: https://discord.gg/uFNUXvFFmg
+<br/>*You can get help faster there!*
 
-**Also read [here](README.md) for basic information.**
+# Docs
+- Basic info: https://github.com/AAndyProgram/SCrawler/blob/main/README.md
+- **GUIDE**: https://github.com/AAndyProgram/SCrawler/wiki/
+- Settings: https://github.com/AAndyProgram/SCrawler/wiki/Settings
+- Discord: https://discord.gg/uFNUXvFFmg
 
 Most of your questions are already answered. All settings, functions, buttons and everything else described in the guide.
 
-Any other questions I will keep in this file.
+# Backup
+I strongly recommend you to **regularly** create backup copies of the settings files. **An [example script](https://github.com/AAndyProgram/SCrawler/blob/main/Tools/ArchiveSCrawlerUsersDataFiles.bat) for this** on GitHub (you **should adapt** it to your environment, and you can use it when [SCrawler is closed](https://github.com/AAndyProgram/SCrawler/wiki/Settings#behavior)).
 
-----
+**This way you'll always have the latest backup of your settings files and can restore it if something goes wrong!**
 
-#### Q: **HOW TO SETUP COOKIES**
+# How to report a problem
-
+1. **Post your problem [here](https://github.com/AAndyProgram/SCrawler/issues) or in the [help channel](https://discord.com/channels/1124032649682493462/1124281838056259614) on our Discord server**
-A: https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies
+2. Attach the **profile URLs or links** that you cannot download.
-
+3. Attach the **LOG** if it exists.
-----
+4. Attach **the environment information** copied from SCrawler (click the top right info button in the main window, then the `Environment` button, then the `Copy` button, and paste the copied text into the message).
-
+5. *Add screenshots to illustrate the problem (**optional**)*
-#### Q: **Does this program have GUI or CLI.**
-
-A: This is a GUI program.
-
-----
-
-#### Q: **Will CLI be added in the future?**
-
-A: NO.
-
-----
-
-#### Q: **I want to add "...." site. How to request.**
-
-<!---A: How to request a new site you can read [here](CONTRIBUTING.md#how-to-request-a-new-site)--->
-**I'm currently not accepting requests to develop new sites.**
-
-----
-
-#### Q: **Site download failed.**
-
-A: Check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set, but nothing works, go to [create a new issue](https://github.com/AAndyProgram/SCrawler/issues). Don't forget to attach the LOG.
-
-**You also can join our Discord server**: https://discord.gg/uFNUXvFFmg
-<br/>*You can get help faster there!*
 
 **ATTENTION! Issues without URLs will be closed without a response!**
 
-----
+# Most frequently questions about SCrawler
 
-#### Q: **I have set credentials but still nothing is downloading**
+**Is something doesn't download, always check the [SITE'S REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements) before asking questions!**
 
-A: Click the `Start downloading` button or press `F5`
+*How to use: find your problem in the list and read the answer.*
 
-----
+## General questions
+- **PROFILES**
+    - I added a profile but **nothing downloaded** :arrow_forward: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. Still nothing works - [report it](#how-to-report-a-problem)!
+    - User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). <u>Don't forget to attach the LOG.</u>
+    - [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user)
+    - How to **add profile** to download :arrow_forward: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
+- How to download **[saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)**
+- **[HOW TO ADD COOKIES](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
+- [How to report a problem](#how-to-report-a-problem)
+- I want you to **add the site** to SCrawler :arrow_forward: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
+- What language is SCrawler written in :arrow_forward: vb.net
+- I don't know vb.net and I can't write a plugin :arrow_forward: you can write a plugin in `C#`
+- I have a suggestion, will it be added :arrow_forward: maybe if it interested me.
+- How to name files using a pattern (e.g. `Site_PostID_Name.jpg`) :arrow_forward: **there is no such functionality and there are no such plans**.
+- **DON'T CHANGE THE DEFAULT SITE SETTINGS UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING!** SCrawler already has all the default settings to work. You only need to add credentials (where [required](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)).
+- My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing** :arrow_forward: restore user settings from [backup](#backup).
+- Installation, update and configuration
+    - How to install: https://github.com/AAndyProgram/SCrawler#installation
+    - How to update: https://github.com/AAndyProgram/SCrawler#updating
+    - What file executes the program: **`SCrawler.exe`**
+    - Where to find binaries: https://github.com/AAndyProgram/SCrawler/releases/latest
+	- [How to build from source](https://github.com/AAndyProgram/SCrawler/blob/main/CONTRIBUTING.md#how-to-build-from-source)
+    - [Video how to configure](#video-how-to-configure)
+- **Antivirus**
+    - **Antivirus detects SCrawler as a virus** :arrow_forward: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
+    - **Antivirus detects gallery-dl as a virus** :arrow_forward: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it**. **But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
 
-#### Q: **Where can I find the release?**
+## Sites questions
 
-A: https://github.com/AAndyProgram/SCrawler/releases/latest
+*How to use: find the site you need in the list and read the answer.*
 
-----
+- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!**
+- **META** (**Instagram**, Threads, Facebook): you need **cookies** and fill in **all fields**
+- **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat.
+- TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download.
+- Porn sites: **COOKIES**!
+- ThisVid: https://github.com/AAndyProgram/SCrawler/wiki/Settings#thisvid-faq
+- **OnlyFans**: cookies + **all fields** + [OF-Scraper (download the correct version that I pointed)](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) & [mp4decrypt](https://www.bento4.com/downloads/) to download DRM protected videos. [OF-Scraper support](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper-support). Also read [this](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans-faq)
+- **JustForFans**: **THE VIDEO ISN'T DOWNLOADING AT THE MOMENT** ([Issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870))
 
-#### Q: **How to run the program?**
+## Other questions
 
-A: Double-click `SCrawler.exe`
+### Does the program remember the last download and check for new posts, downloading only new posts, or does the program download the entire profile every time
+The program stored posts IDs in users' folders. For the first time, the program downloads the entire profile. All subsequent times the program will check for new posts and download **only new posts**!
 
-----
+### Does this program have a GUI or CLI, and will a CLI be added in the future
+This is a GUI program and **NO**, <u>CLI will not be added</u>
 
-#### Q: **Where to find binaries?**
+### How to remove the label
+There is no functionality to remove an individual label. You can open the `Labels.txt` file in the program settings folder and delete any label you want. You also can delete this file (`Labels.txt`). In this case, when SCrawler is launched, the list of labels will be populated only with existing labels (from the user data files).
 
-A: https://github.com/AAndyProgram/SCrawler/releases/latest
+### How to remove a user from the blacklist
+Just add that user back to the program. In the dialog box that opens, click the `Add and remove from blacklist` button.
 
-----
+### You lost me. Your program is too complicated.
+**I'm fine with that**. If the program is too complicated for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I develop SCrawler for myself and publish it on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm okay with it.
 
-#### Q: **Does the program remember the last download and check for new posts, downloading only new posts? Or does the program download the entire profile every time?**
+### Add a step-by-step guide or video on how to use the program
+**NO!** The guide fully covers all the functionality of SCrawler! If you don't respect my work, I don't waste my time. If you want, you can create a video tutorial and send it to me. Then I'll add it. All options and their purposes are described on the wiki. The wiki also contains a description of all the settings and how to configure them. For complex settings there is a step-by-step guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I've developed a program with an intuitive interface. There is a `Settings` button, download buttons, a context menu that appears when you right-click on a user, and other controls. Anyone can use it.
 
-A: The program stored posts IDs in users' folders. For the first time, the program downloads the entire profile. All subsequent times the program will check for new posts and download **only new posts**!
+**There is already a [video](#video-how-to-configure) example of how to configure a site.**
 
-----
+# Video how to configure
-
-#### Q: **How to redownload all data**
-
-A: https://github.com/AAndyProgram/SCrawler/wiki#redownload-user
-
-----
-
-#### Q: **How to remove the label**
-
-A: There is no functionality to remove an individual label. You can open the `Labels.txt` file in the program settings folder and delete any label you want. You also can delete this file (`Labels.txt`). In this case, when the program starts, the list of labels list will be updated with only existing labels (from the user data files).
-
-----
-
-#### Q: **How to remove a user from the blacklist**
-
-A: Just add that user back to the program. In the dialog box that opens, click on the `Add and remove from blacklist` button.
-
-----
-
-#### Q: **Why don't you answer how it works**
-
-A: Because **I don't want to**. I don't want to waste my time explaining things that are already covered in the **[GUIDE](https://github.com/AAndyProgram/SCrawler/wiki)**! If you didn't bother to read the guide, why would I waste my time?! ALL FUNCTIONALITY IS DESCRIBED IN THE GUIDE. Before publishing a new release, I update the guide. If you don't respect my work, I don't waste my time.
-
-----
-
-#### Q: **You lost me. Your program is too complicated.**
-
-A: **I'm fine with that**. If the program is difficult for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I am develop SCrawler for myself and publish on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm fine.
-
-----
-
-#### Q: **I can't configure something**
-
-A: I can only [suggest](#q-you-lost-me-your-program-is-too-complicated) you find another (easier) program.
-
-----
-
-#### Q: **Can you add a step-by-step guide or video on how to use the program?**
-
-A: **NO!** The guide fully covers all the functionality of SCrawler! If you don't respect my work, I don't waste my time. If you want, you can create a video tutorial and send it to me. Then I add it. All options and what each option does described on the wiki. The wiki also contains a description of all settings and how-to configure them. For complex settings, there is a steep-by-steep guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I have developed a program with an intuitive interface. There is a Settings button, download buttons, a context menu that drops down when a user is clicked, and other controls. Anyone can use it.
 
 **The following video was recorded by a user who loves SCrawler and demonstrates how to add credentials using Instagram as an example:**
 

README.md

@@ -82,7 +82,7 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo
 - **Instagram**
 - **Threads**
 - **Facebook**
-- JustForFans *(partial support)*[^1]
+- JustForFans *(partial support) ([video issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870))*[^1]
 - Mastodon *(out of support)*
 - TikTok
 - RedGifs

SCrawler.YouTube/Controls/VideoOptionsForm.vb

@@ -32,6 +32,7 @@ Namespace API.YouTube.Controls
         Private Initialization As Boolean = True
         Private ReadOnly InheritsFromContainer As Boolean
         Private ReadOnly M3U8Files As List(Of SFile)
+        Friend Property UseCookies As Boolean = False
         Private ReadOnly Property M3U8FilesFull As List(Of SFile)
             Get
                 Return ListAddList(Nothing, M3U8Files, LAP.NotContainsOnly).ListAddValue(CMB_PLS.Text, LAP.NotContainsOnly)
@@ -65,6 +66,7 @@ Namespace API.YouTube.Controls
             CNT_PROCESSOR = New TableControlsProcessor(TP_CONTROLS)
             Me.InheritsFromContainer = InheritsFromContainer
             MyFieldsChecker = New FieldsChecker
+            UseCookies = MyYouTubeSettings.DefaultUseCookies
         End Sub
 #End Region
 #Region "Form handlers"
@@ -227,7 +229,7 @@ Namespace API.YouTube.Controls
                 Dim data As IEnumerable(Of Control)
 
                 If .HasElements Then
-                    data = .Elements.Select(Function(ee) New MediaItem(ee, True) With {.Dock = DockStyle.Fill, .Checked = ee.Checked})
+                    data = .Elements.Select(Function(ee) New MediaItem(ee, True) With {.Dock = DockStyle.Fill, .Checked = ee.Checked, .UseCookies = UseCookies})
                 Else
                     data = (From m As MediaObject In .Self.MediaObjects
                             Where m.Type = __contentType

SCrawler.YouTube/Downloader/MediaItem.vb

@@ -437,7 +437,7 @@ Namespace DownloadObjects.STDownloader
                     Else
                         RaiseEvent BeforeOpenEditor(Me, MyContainer)
                     End If
-                    Using f As New VideoOptionsForm(MyContainer, initProtected Or isFull)
+                    Using f As New VideoOptionsForm(MyContainer, initProtected Or isFull) With {.UseCookies = UseCookies}
                         f.ShowDialog()
                         .Protected = IIf(f.DialogResult = DialogResult.OK, True, initProtected)
                     End Using
@@ -465,12 +465,12 @@ Namespace DownloadObjects.STDownloader
             If Not MyContainer Is Nothing Then
                 Dim f As Form = Nothing
                 Select Case MyContainer.ObjectType
-                    Case Base.YouTubeMediaType.Single : f = New VideoOptionsForm(MyContainer, True)
+                    Case Base.YouTubeMediaType.Single : f = New VideoOptionsForm(MyContainer, True) With {.UseCookies = UseCookies}
                     Case Base.YouTubeMediaType.Channel, Base.YouTubeMediaType.PlayList
                         If MyContainer.IsMusic Then
                             f = New MusicPlaylistsForm(MyContainer)
                         Else
-                            f = New VideoOptionsForm(MyContainer, True)
+                            f = New VideoOptionsForm(MyContainer, True) With {.UseCookies = UseCookies}
                         End If
                 End Select
                 If Not f Is Nothing Then

SCrawler.YouTube/Downloader/VideoListForm.vb

@@ -132,7 +132,8 @@ Namespace DownloadObjects.STDownloader
 #End Region
 #Region "Controls"
         Protected Sub ControlCreateAndAdd(ByVal Container As IYouTubeMediaContainer, Optional ByVal DisableDownload As Boolean = False,
-                                          Optional ByVal PerformClick As Boolean = True, Optional ByVal IsLoading As Boolean = False)
+                                          Optional ByVal PerformClick As Boolean = True, Optional ByVal IsLoading As Boolean = False,
+                                          Optional ByVal UseCookies As Boolean = False)
             ControlInvokeFast(TP_CONTROLS, Sub()
                                                With TP_CONTROLS
                                                    .SuspendLayout()
@@ -142,7 +143,7 @@ Namespace DownloadObjects.STDownloader
                                                    .RowStyles.Insert(0, New RowStyle(SizeType.Absolute, 60))
                                                    .RowCount = .RowStyles.Count
                                                    OffsetControls(0, True)
-                                                   Dim cnt As New MediaItem(Container) With {.Dock = DockStyle.Fill, .Margin = New Padding(0)}
+                                                   Dim cnt As New MediaItem(Container) With {.Dock = DockStyle.Fill, .Margin = New Padding(0), .UseCookies = UseCookies}
                                                    AddHandler cnt.FileDownloaded, AddressOf MediaControl_FileDownloaded
                                                    AddHandler cnt.Removal, AddressOf MediaControl_Removal
                                                    AddHandler cnt.DownloadAgain, AddressOf MediaControl_DownloadAgain
@@ -333,19 +334,19 @@ Namespace DownloadObjects.STDownloader
                     If Not c Is Nothing Then
                         Dim f As Form
                         Select Case c.ObjectType
-                            Case YouTubeMediaType.Single : f = New VideoOptionsForm(c)
+                            Case YouTubeMediaType.Single : f = New VideoOptionsForm(c) With {.UseCookies = useCookies}
                             Case YouTubeMediaType.Channel, YouTubeMediaType.PlayList
                                 If c.IsMusic Then
                                     f = New MusicPlaylistsForm(c)
                                 Else
-                                    f = New VideoOptionsForm(c)
+                                    f = New VideoOptionsForm(c) With {.UseCookies = useCookies}
                                 End If
                             Case Else : c.Dispose() : Throw New ArgumentException($"Object type {c.ObjectType} not implemented", "IYouTubeMediaContainer.ObjectType")
                         End Select
                         If Not f Is Nothing Then
                             If TypeOf f Is IDesignXMLContainer Then DirectCast(f, IDesignXMLContainer).DesignXML = DesignXML
                             f.ShowDialog()
-                            If f.DialogResult = DialogResult.OK AndAlso ValidateContainerURL(c) Then ControlCreateAndAdd(c, disableDown)
+                            If f.DialogResult = DialogResult.OK AndAlso ValidateContainerURL(c) Then ControlCreateAndAdd(c, disableDown,,, useCookies)
                             f.Dispose()
                         End If
                     End If

SCrawler.YouTube/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2024.7.24.0")>
+<Assembly: AssemblyVersion("2024.8.10.0")>
-<Assembly: AssemblyFileVersion("2024.7.24.0")>
+<Assembly: AssemblyFileVersion("2024.8.10.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler.YouTubeDownloader/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2024.7.24.0")>
+<Assembly: AssemblyVersion("2024.8.10.0")>
-<Assembly: AssemblyFileVersion("2024.7.24.0")>
+<Assembly: AssemblyFileVersion("2024.8.10.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler/API/Base/UserDataBase.vb

@@ -410,9 +410,7 @@ Namespace API.Base
         End Function
         Friend Overridable Sub SetPicture(ByVal f As SFile) Implements IUserData.SetPicture
             Try
-                If f.Exists Then
+                If f.Exists Then UserImage.NewUserPicture(f, MyFile)
-                    Using p As New UserImage(f, MyFile) : p.Save() : End Using
-                End If
             Catch
             End Try
         End Sub
@@ -451,11 +449,7 @@ BlockPictureScan:
                                                      New ErrorsDescriber(EDP.ReturnValue) With {
                                                          .ReturnValue = New List(Of SFile),
                                                          .ReturnValueExists = True}).FirstOrDefault
-            If NewPicFile.Exists Then
+            If NewPicFile.Exists Then p = UserImage.NewUserPicture(NewPicFile, MyFile,, True) : GoTo BlockReturn
-                p = New UserImage(NewPicFile, MyFile)
-                p.Save()
-                GoTo BlockReturn
-            End If
 BlockDeletePictureFolder:
             On Error GoTo BlockReturn
             If DelPath Then
@@ -654,6 +648,7 @@ BlockNullPicture:
         End Sub
         Protected ReadOnly _TempMediaList As List(Of UserMedia)
         Protected ReadOnly _TempPostsList As List(Of String)
+        Private ReadOnly _MD5List As List(Of String)
         Friend Function GetLastImageAddress() As SFile
             If _ContentList.Count > 0 Then
                 Return _ContentList.LastOrDefault(Function(c) c.Type = UTypes.Picture And Not c.File.IsEmptyString And Not c.File.Extension = "gif").File
@@ -679,6 +674,7 @@ BlockNullPicture:
         Protected MyFileSettings As SFile
         Protected MyFileData As SFile
         Protected MyFilePosts As SFile
+        Private MyMD5File As SFile
         Friend Overridable Property FileExists As Boolean = False Implements IUserData.FileExists
         Friend Overridable Property DataMerging As Boolean
             Get
@@ -856,6 +852,7 @@ BlockNullPicture:
             LatestData = New List(Of UserMedia)
             _TempMediaList = New List(Of UserMedia)
             _TempPostsList = New List(Of String)
+            _MD5List = New List(Of String)
             Labels = New List(Of String)
             UserUpdatedEventHandlers = New List(Of IUserData.UserUpdatedEventHandler)
             UserDownloadStateChangedEventHandlers = New List(Of UserDownloadStateChangedEventHandler)
@@ -1037,6 +1034,8 @@ BlockNullPicture:
                     If _ContentList.Count > 0 Then x.AddRange(_ContentList)
                     x.Save(MyFileData)
                 End Using
+                If Not MyMD5File.IsEmptyString And _MD5List.Count > 0 Then _
+                   TextSaver.SaveTextToFile(_MD5List.ListToString(Environment.NewLine), MyMD5File, True,, EDP.None)
             Catch ex As Exception
                 LogError(ex, "history saving error")
             End Try
@@ -1131,6 +1130,7 @@ BlockNullPicture:
             TokenPersonal = Nothing
             ProgressPre.Reset()
             UpdateDataFiles()
+            _MD5Loaded = False
             _DownloadInProgress = True
             _DescriptionChecked = False
             _DescriptionEveryTime = Settings.UpdateUserDescriptionEveryTime
@@ -1212,7 +1212,7 @@ BlockNullPicture:
                 ProgressPre.Done()
                 ThrowAny(Token)
 
-                If UseMD5Comparison And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
+                If RemoveExistingDuplicates And Not IsSubscription Then ValidateMD5(Token) : ProgressPre.Done() : ThrowAny(Token)
 
                 If _TempPostsList.Count > 0 And Not DownloadMissingOnly And Not __isChannelsSupport Then
                     If _TempPostsList.Count > 1000 Then _TempPostsList.ListAddList(_TempPostsList.ListTake(-2, 1000, EDP.ReturnValue).ListReverse, LAP.ClearBeforeAdd)
@@ -1315,6 +1315,11 @@ BlockNullPicture:
                 MyFilePosts = MyFileSettings
                 MyFilePosts.Name &= "_Posts"
                 MyFilePosts.Extension = "txt"
+                If Not IsSavedPosts Then
+                    MyMD5File = MyFileSettings
+                    MyMD5File.Name &= "_MD5"
+                    MyMD5File.Extension = "txt"
+                End If
             Else
                 Throw New ArgumentNullException("User.File", "User file not detected")
             End If
@@ -1438,81 +1443,94 @@ BlockNullPicture:
         End Sub
 #End Region
 #Region "MD5 support"
-        Protected Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
+        Private Const VALIDATE_MD5_ERROR As String = "VALIDATE_MD5_ERROR"
         Friend Property UseMD5Comparison As Boolean = False
         Protected Property StartMD5Checked As Boolean = False
         Friend Property RemoveExistingDuplicates As Boolean = False
-        Protected Overridable Sub ValidateMD5(ByVal Token As CancellationToken)
+        Private ReadOnly ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
+        Private _MD5Loaded As Boolean = False
+        Private Sub LoadMD5()
+            Try
+                If Not _MD5Loaded Then
+                    _MD5Loaded = True
+                    _MD5List.Clear()
+                    If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(c) c.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
+                    If MyMD5File.Exists Then _MD5List.ListAddList(MyMD5File.GetLines, LAP.NotContainsOnly, EDP.ThrowException)
+                End If
+            Catch ex As Exception
+                ErrorsDescriber.Execute(EDP.SendToLog, ex, "LoadMD5")
+            End Try
+        End Sub
+        Private Function ValidateMD5_GetMD5(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
+            Try
+                Dim ImgFormat As Imaging.ImageFormat = Nothing
+                Dim hash$ = String.Empty
+                Dim __isGif As Boolean = False
+                If __data.Type = UTypes.GIF Then
+                    ImgFormat = Imaging.ImageFormat.Gif
+                    __isGif = True
+                ElseIf Not __data.File.IsEmptyString Then
+                    ImgFormat = GetImageFormat(__data.File)
+                End If
+                If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
+                If IsUrl And Not __isGif Then
+                    hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
+                ElseIf IsUrl And __isGif Then
+                    hash = ByteArrayToString(GetMD5FromBytes(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ErrMD5))
+                Else
+                    hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
+                End If
+                If hash.IsEmptyString And Not __isGif Then
+                    If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
+                    If IsUrl Then
+                        hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
+                    Else
+                        hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
+                    End If
+                End If
+                Return hash
+            Catch
+                Return String.Empty
+            End Try
+        End Function
+        Private Sub ValidateMD5(ByVal Token As CancellationToken)
             Try
                 Dim missingMD5 As Predicate(Of UserMedia) = Function(d) (d.Type = UTypes.GIF Or d.Type = UTypes.Picture) And d.MD5.IsEmptyString
-                If UseMD5Comparison And _TempMediaList.Exists(missingMD5) Then
+                If RemoveExistingDuplicates Then
+                    RemoveExistingDuplicates = False
+                    _ForceSaveUserInfo = True
+                    LoadMD5()
                     Dim i%
                     Dim itemsCount% = 0
                     Dim limit% = If(DownloadTopCount, 0)
                     Dim data As UserMedia = Nothing
-                    Dim hashList As New Dictionary(Of String, SFile)
                     Dim f As SFile
-                    Dim ErrMD5 As New ErrorsDescriber(EDP.ReturnValue)
+
-                    Dim __getMD5 As Func(Of UserMedia, Boolean, String) =
-                        Function(ByVal __data As UserMedia, ByVal IsUrl As Boolean) As String
-                            Try
-                                Dim ImgFormat As Imaging.ImageFormat = Nothing
-                                Dim hash$ = String.Empty
-                                Dim __isGif As Boolean = False
-                                If __data.Type = UTypes.GIF Then
-                                    ImgFormat = Imaging.ImageFormat.Gif
-                                    __isGif = True
-                                ElseIf Not __data.File.IsEmptyString Then
-                                    ImgFormat = GetImageFormat(__data.File)
-                                End If
-                                If ImgFormat Is Nothing Then ImgFormat = Imaging.ImageFormat.Jpeg
-                                If IsUrl And Not __isGif Then
-                                    hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
-                                ElseIf IsUrl And __isGif Then
-                                    hash = ByteArrayToString(GetMD5FromBytes(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ErrMD5))
-                                Else
-                                    hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
-                                End If
-                                If hash.IsEmptyString And Not __isGif Then
-                                    If ImgFormat Is Imaging.ImageFormat.Jpeg Then ImgFormat = Imaging.ImageFormat.Png Else ImgFormat = Imaging.ImageFormat.Jpeg
-                                    If IsUrl Then
-                                        hash = ByteArrayToString(GetMD5(SFile.GetBytesFromNet(__data.URL.IfNullOrEmpty(__data.URL_BASE), ErrMD5), ImgFormat, ErrMD5))
-                                    Else
-                                        hash = ByteArrayToString(GetMD5(SFile.GetBytes(__data.File, ErrMD5), ImgFormat, ErrMD5))
-                                    End If
-                                End If
-                                Return hash
-                            Catch
-                                Return String.Empty
-                            End Try
-                        End Function
                     If Not StartMD5Checked Then
                         StartMD5Checked = True
-                        If _ContentList.Exists(missingMD5) Then
+                        Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
-                            Dim existingFiles As List(Of SFile) = SFile.GetFiles(MyFileSettings.CutPath, "*.jpg|*.jpeg|*.png|*.gif",, EDP.ReturnValue).ListIfNothing
+                        Dim eIndx%
-                            Dim eIndx%
+                        Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
-                            Dim eFinder As Predicate(Of SFile) = Function(ff) ff.File = data.File.File
+
-                            If RemoveExistingDuplicates Then
+                        If existingFiles.Count > 0 Then
-                                RemoveExistingDuplicates = False
+                            Dim h$
-                                _ForceSaveUserInfo = True
+                            ProgressPre.ChangeMax(existingFiles.Count)
-                                If existingFiles.Count > 0 Then
+                            For i = existingFiles.Count - 1 To 0 Step -1
-                                    Dim h$
+                                ProgressPre.Perform()
-                                    ProgressPre.ChangeMax(existingFiles.Count)
+                                h = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(i)}, False)
-                                    For i = existingFiles.Count - 1 To 0 Step -1
+                                If Not h.IsEmptyString Then
-                                        ProgressPre.Perform()
+                                    If _MD5List.Contains(h) Then
-                                        h = __getMD5(New UserMedia With {.File = existingFiles(i)}, False)
+                                        MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate)"
-                                        If Not h.IsEmptyString Then
+                                        existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
-                                            If hashList.ContainsKey(h) Then
+                                        existingFiles.RemoveAt(i)
-                                                MyMainLOG = $"{ToStringForLog()}: Removed image [{existingFiles(i).File}] (duplicate of [{hashList(h).File}])"
+                                    Else
-                                                existingFiles(i).Delete(SFO.File, SFODelete.DeleteToRecycleBin, ErrMD5)
+                                        _MD5List.Add(h)
-                                                existingFiles.RemoveAt(i)
+                                    End If
-                                            Else
-                                                hashList.Add(h, existingFiles(i))
-                                            End If
-                                        End If
-                                    Next
                                 End If
-                            End If
+                            Next
+                        End If
+
+                        If _ContentList.Count > 0 AndAlso _ContentList.Exists(missingMD5) Then
                             ProgressPre.ChangeMax(_ContentList.Count)
                             For i = 0 To _ContentList.Count - 1
                                 data = _ContentList(i)
@@ -1522,61 +1540,34 @@ BlockNullPicture:
                                         ThrowAny(Token)
                                         eIndx = existingFiles.FindIndex(eFinder)
                                         If eIndx >= 0 Then
-                                            data.MD5 = __getMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
+                                            data.MD5 = ValidateMD5_GetMD5(New UserMedia With {.File = existingFiles(eIndx)}, False)
                                             If Not data.MD5.IsEmptyString Then _ContentList(i) = data : _ForceSaveUserData = True
                                         End If
                                     End If
                                     existingFiles.RemoveAll(eFinder)
                                 End If
                             Next
-                            If existingFiles.Count > 0 Then
-                                ProgressPre.ChangeMax(existingFiles.Count)
-                                For i = 0 To existingFiles.Count - 1
-                                    f = existingFiles(i)
-                                    ProgressPre.Perform()
-                                    data = New UserMedia(f.File) With {
-                                        .State = UStates.Downloaded,
-                                        .Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
-                                        .File = f
-                                    }
-                                    ThrowAny(Token)
-                                    data.MD5 = __getMD5(data, False)
-                                    If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
-                                Next
-                                existingFiles.Clear()
-                            End If
                         End If
-                    End If
 
-                    If _ContentList.Count > 0 Then
+                        If existingFiles.Count > 0 Then
-                        With _ContentList.Select(Function(d) d.MD5)
+                            ProgressPre.ChangeMax(existingFiles.Count)
-                            If .ListExists Then .ToList.ForEach(Sub(md5value) _
+                            For i = 0 To existingFiles.Count - 1
-                               If Not md5value.IsEmptyString AndAlso Not hashList.ContainsKey(md5value) Then hashList.Add(md5value, New SFile))
+                                f = existingFiles(i)
-                        End With
+                                ProgressPre.Perform()
-                    End If
+                                data = New UserMedia(f.File) With {
-
+                                    .State = UStates.Downloaded,
-                    ProgressPre.ChangeMax(_TempMediaList.Count)
+                                    .Type = IIf(f.Extension = "gif", UTypes.GIF, UTypes.Picture),
-                    For i = _TempMediaList.Count - 1 To 0 Step -1
+                                    .File = f
-                        ProgressPre.Perform()
+                                }
-                        If limit > 0 And itemsCount >= limit Then
-                            _TempMediaList.RemoveAt(i)
-                        Else
-                            data = _TempMediaList(i)
-                            If missingMD5(data) Then
                                 ThrowAny(Token)
-                                data.MD5 = __getMD5(data, True)
+                                data.MD5 = ValidateMD5_GetMD5(data, False)
-                                If Not data.MD5.IsEmptyString Then
+                                If Not data.MD5.IsEmptyString Then _ContentList.Add(data) : _ForceSaveUserData = True
-                                    If hashList.ContainsKey(data.MD5) Then
+                            Next
-                                        _TempMediaList.RemoveAt(i)
+                            existingFiles.Clear()
-                                    Else
-                                        hashList.Add(data.MD5, New SFile)
-                                        _TempMediaList(i) = data
-                                        itemsCount += 1
-                                    End If
-                                End If
-                            End If
                         End If
-                    Next
+                    End If
+
+                    If _ContentList.Count > 0 Then _MD5List.ListAddList(_ContentList.Select(Function(d) d.MD5), LAP.NotContainsOnly, EDP.ReturnValue)
                 End If
             Catch iex As ArgumentOutOfRangeException When Disposed
             Catch ex As Exception
@@ -1614,6 +1605,7 @@ BlockNullPicture:
                 Source.Progress.Done()
             End Sub
         End Class
+        Protected Const VideoFolderName As String = "Video"
         Protected Sub DownloadContentDefault(ByVal Token As CancellationToken)
             Try
                 Dim i%
@@ -1622,6 +1614,7 @@ BlockNullPicture:
                 If _ContentNew.Count > 0 Then
                     _ContentNew.RemoveAll(Function(c) c.URL.IsEmptyString)
                     If _ContentNew.Count > 0 Then
+                        If UseMD5Comparison Then LoadMD5()
                         MyFile.Exists(SFO.Path)
                         Dim MissingErrorsAdd As Boolean = Settings.AddMissingErrorsToLog
                         Dim MyDir$ = DownloadContentDefault_GetRootDir()
@@ -1630,6 +1623,7 @@ BlockNullPicture:
                         Dim __interrupt As Boolean
                         Dim f As SFile
                         Dim v As UserMedia
+                        Dim __fileDeleted As Boolean
                         Dim fileNumProvider As SFileNumbers = SFileNumbers.Default
                         Dim __deleteFile As Action(Of SFile, String) = Sub(ByVal FileToDelete As SFile, ByVal FileUrl As String)
                                                                            Try
@@ -1641,9 +1635,21 @@ BlockNullPicture:
                                                                                ErrorsDescriber.Execute(EDP.SendToLog, file_del_ex)
                                                                            End Try
                                                                        End Sub
+                        Dim updateDownCount As Action = Sub()
+                                                            Dim __n% = IIf(__fileDeleted, -1, 1)
+                                                            If __isVideo Then
+                                                                v.Type = UTypes.Video
+                                                                DownloadedVideos(False) += __n
+                                                            ElseIf v.Type = UTypes.GIF Then
+                                                                DownloadedPictures(False) += __n
+                                                            Else
+                                                                v.Type = UTypes.Picture
+                                                                DownloadedPictures(False) += __n
+                                                            End If
+                                                        End Sub
 
                         Using w As New OptionalWebClient(Me)
-                            If vsf Then CSFileP($"{MyDir}\Video\").Exists(SFO.Path)
+                            If vsf Then CSFileP($"{MyDir}\{VideoFolderName}\").Exists(SFO.Path)
                             Progress.Maximum += _ContentNew.Count
                             If IsSingleObjectDownload Then
                                 If _ContentNew.Count = 1 And _ContentNew(0).Type = UTypes.Video Then
@@ -1671,6 +1677,8 @@ BlockNullPicture:
 
                                 If v.URL_BASE.IsEmptyString Then v.URL_BASE = v.URL
 
+                                __fileDeleted = False
+
                                 If Not f.IsEmptyString And Not v.URL.IsEmptyString Then
                                     Try
                                         __isVideo = v.Type = UTypes.Video Or f.Extension = "mp4" Or v.Type = UTypes.m3u8
@@ -1691,7 +1699,7 @@ BlockNullPicture:
                                         End If
                                         If __isVideo And vsf Then
                                             If v.SpecialFolder.IsEmptyString OrElse Not v.SpecialFolder.EndsWith("*") Then
-                                                f.Path = $"{f.PathWithSeparator}Video"
+                                                f.Path = $"{f.PathWithSeparator}{VideoFolderName}"
                                                 If Not v.SpecialFolder.IsEmptyString Then f.Exists(SFO.Path)
                                             End If
                                         End If
@@ -1715,19 +1723,26 @@ BlockNullPicture:
                                             End If
                                         End If
 
-                                        If __isVideo Then
+                                        updateDownCount()
-                                            v.Type = UTypes.Video
-                                            DownloadedVideos(False) += 1
-                                        ElseIf v.Type = UTypes.GIF Then
-                                            DownloadedPictures(False) += 1
-                                        Else
-                                            v.Type = UTypes.Picture
-                                            DownloadedPictures(False) += 1
-                                        End If
 
                                         v.File = ChangeFileNameByProvider(f, v)
                                         v.State = UStates.Downloaded
                                         DownloadContentDefault_PostProcessing(v, f, Token)
+                                        If UseMD5Comparison And (v.Type = UTypes.GIF Or v.Type = UTypes.Picture) Then
+                                            If v.File.Exists Then
+                                                v.MD5 = ValidateMD5_GetMD5(v, False)
+                                                If Not v.MD5.IsEmptyString Then
+                                                    If _MD5List.Contains(v.MD5) Then
+                                                        __fileDeleted = v.File.Delete(SFO.File, SFODelete.DeletePermanently, EDP.ReturnValue)
+                                                        If __fileDeleted Then dCount -= 1 : updateDownCount()
+                                                    Else
+                                                        _MD5List.Add(v.MD5)
+                                                    End If
+                                                End If
+                                            Else
+                                                dCount -= 1
+                                            End If
+                                        End If
                                         dCount += 1
                                     Catch woex As OperationCanceledException When Token.IsCancellationRequested
                                         __deleteFile.Invoke(f, v.URL_BASE)
@@ -1745,7 +1760,7 @@ BlockNullPicture:
                                 Else
                                     v.State = UStates.Skipped
                                 End If
-                                _ContentNew(i) = v
+                                If Not __fileDeleted Then _ContentNew(i) = v
                                 If DownloadTopCount.HasValue AndAlso dCount >= DownloadTopCount.Value Then
                                     Progress.Perform(_ContentNew.Count - dTotal)
                                     Exit Sub
@@ -2240,6 +2255,7 @@ BlockNullPicture:
                     LatestData.Clear()
                     _TempMediaList.Clear()
                     _TempPostsList.Clear()
+                    _MD5List.Clear()
                     TokenPersonal = Nothing
                     If Not ProgressPre Is Nothing Then ProgressPre.Reset() : ProgressPre.Dispose()
                     If Not Responser Is Nothing Then Responser.Dispose()

SCrawler/API/Instagram/EditorExchangeOptions.vb

@@ -9,6 +9,7 @@
 Imports SCrawler.Plugin.Attributes
 Namespace API.Instagram
     Friend Class EditorExchangeOptions
+#Region "Download"
         <PSetting(Caption:="Get timeline", ToolTip:="Download user timeline")>
         Friend Property GetTimeline As Boolean
         <PSetting(Caption:="Get reels", ToolTip:="Download user reels")>
@@ -19,6 +20,21 @@ Namespace API.Instagram
         Friend Property GetStoriesUser As Boolean
         <PSetting(Caption:="Get tagged posts", ToolTip:="Download user tagged posts")>
         Friend Property GetTagged As Boolean
+#End Region
+#Region "Extract image"
+        <PSetting(Caption:="Extract image from video: timeline")>
+        Friend Property GetTimeline_VideoPic As Boolean
+        <PSetting(Caption:="Extract image from video: reels")>
+        Friend Property GetReels_VideoPic As Boolean
+        <PSetting(Caption:="Extract image from video: stories")>
+        Friend Property GetStories_VideoPic As Boolean
+        <PSetting(Caption:="Extract image from video: stories: user")>
+        Friend Property GetStoriesUser_VideoPic As Boolean
+        <PSetting(Caption:="Extract image from video: tagged posts")>
+        Friend Property GetTagged_VideoPic As Boolean
+#End Region
+        <PSetting(Caption:="Place the extracted image into the video folder")>
+        Friend Property PutImageVideoFolder As Boolean
         Friend Sub New(ByVal u As UserData)
             With u
                 GetTimeline = .GetTimeline
@@ -26,6 +42,14 @@ Namespace API.Instagram
                 GetStories = .GetStories
                 GetStoriesUser = .GetStoriesUser
                 GetTagged = .GetTaggedData
+
+                GetTimeline_VideoPic = .GetTimeline_VideoPic
+                GetReels_VideoPic = .GetReels_VideoPic
+                GetStories_VideoPic = .GetStories_VideoPic
+                GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
+                GetTagged_VideoPic = .GetTaggedData_VideoPic
+
+                PutImageVideoFolder = .PutImageVideoFolder
             End With
         End Sub
         Friend Sub New(ByVal s As SiteSettings)
@@ -35,6 +59,14 @@ Namespace API.Instagram
                 GetStories = CBool(.GetStories.Value)
                 GetStoriesUser = CBool(.GetStoriesUser.Value)
                 GetTagged = CBool(.GetTagged.Value)
+
+                GetTimeline_VideoPic = CBool(.GetTimeline_VideoPic.Value)
+                GetReels_VideoPic = CBool(.GetReels_VideoPic.Value)
+                GetStories_VideoPic = CBool(.GetStories_VideoPic.Value)
+                GetStoriesUser_VideoPic = CBool(.GetStoriesUser_VideoPic.Value)
+                GetTagged_VideoPic = CBool(.GetTagged_VideoPic.Value)
+
+                PutImageVideoFolder = CBool(.PutImageVideoFolder.Value)
             End With
         End Sub
     End Class

SCrawler/API/Instagram/SiteSettings.vb

@@ -57,6 +57,7 @@ Namespace API.Instagram
 #End Region
 #Region "Categories"
         Private Const CAT_DOWN As String = "Download data"
+        Private Const CAT_UserDefs_VIDEO As String = DN.CAT_UserDefs & ": extract image from video"
 #End Region
 #Region "Authorization properties"
         Friend Const Header_IG_APP_ID As String = "x-ig-app-id"
@@ -187,14 +188,28 @@ Namespace API.Instagram
         Private ReadOnly Property SleepTimerOnPostsLimitProvider As IFormatProvider
         <PropertyOption(ControlText:="Get timeline", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(23), PClonable>
         Friend ReadOnly Property GetTimeline As PropertyValue
+        <PropertyOption(ControlText:="From timeline", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(23), PClonable>
+        Friend ReadOnly Property GetTimeline_VideoPic As PropertyValue
         <PropertyOption(ControlText:="Get reels", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(24), PClonable>
         Friend ReadOnly Property GetReels As PropertyValue
+        <PropertyOption(ControlText:="From reels", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(24), PClonable>
+        Friend ReadOnly Property GetReels_VideoPic As PropertyValue
         <PropertyOption(ControlText:="Get stories", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(25), PClonable>
         Friend ReadOnly Property GetStories As PropertyValue
+        <PropertyOption(ControlText:="From stories", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(25), PClonable>
+        Friend ReadOnly Property GetStories_VideoPic As PropertyValue
         <PropertyOption(ControlText:="Get stories: user", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(26), PClonable>
         Friend ReadOnly Property GetStoriesUser As PropertyValue
-        <PropertyOption(ControlText:="Get tagged photos", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(27), PClonable>
+        <PropertyOption(ControlText:="From stories: user", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(26), PClonable>
+        Friend ReadOnly Property GetStoriesUser_VideoPic As PropertyValue
+        <PropertyOption(ControlText:="Get tagged posts", ControlToolTip:="Default value for new users", Category:=DN.CAT_UserDefs), PXML, ControlNumber(27), PClonable>
         Friend ReadOnly Property GetTagged As PropertyValue
+        <PropertyOption(ControlText:="From tagged posts", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(27), PClonable>
+        Friend ReadOnly Property GetTagged_VideoPic As PropertyValue
+        <PropertyOption(ControlText:="From saved posts", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(28), PClonable>
+        Friend ReadOnly Property GetSavedPosts_VideoPic As PropertyValue
+        <PropertyOption(ControlText:="Place the extracted image into the video folder", ControlToolTip:="Default value for new users", Category:=CAT_UserDefs_VIDEO), PXML, ControlNumber(29), PClonable>
+        Friend ReadOnly Property PutImageVideoFolder As PropertyValue
         <PropertyOption(ControlText:="Tagged notify limit",
                         ControlToolTip:="If the number of tagged posts exceeds this number you will be notified." & vbCr &
                         "-1 to disable"), PXML, ControlNumber(27), PClonable>
@@ -203,19 +218,19 @@ Namespace API.Instagram
         Private ReadOnly Property TaggedNotifyLimitProvider As IFormatProvider
 #End Region
 #Region "Download ready"
-        <PropertyOption(ControlText:="Download timeline", ControlToolTip:="Download timeline", Category:=CAT_DOWN), PXML, ControlNumber(10), PClonable>
+        <PropertyOption(ControlText:="Download timeline", Category:=CAT_DOWN), PXML, ControlNumber(10), PClonable>
         Friend ReadOnly Property DownloadTimeline As PropertyValue
         <PXML> Private ReadOnly Property DownloadTimeline_Def As PropertyValue
-        <PropertyOption(ControlText:="Download reels", ControlToolTip:="Download reels", Category:=CAT_DOWN), PXML, ControlNumber(11), PClonable>
+        <PropertyOption(ControlText:="Download reels", Category:=CAT_DOWN), PXML, ControlNumber(11), PClonable>
         Friend ReadOnly Property DownloadReels As PropertyValue
         <PXML> Private ReadOnly Property DownloadReels_Def As PropertyValue
-        <PropertyOption(ControlText:="Download stories", ControlToolTip:="Download stories", Category:=CAT_DOWN), PXML, ControlNumber(12), PClonable>
+        <PropertyOption(ControlText:="Download stories", Category:=CAT_DOWN), PXML, ControlNumber(12), PClonable>
         Friend ReadOnly Property DownloadStories As PropertyValue
         <PXML> Private ReadOnly Property DownloadStories_Def As PropertyValue
-        <PropertyOption(ControlText:="Download stories: user", ControlToolTip:="Download stories (user)", Category:=CAT_DOWN), PXML, ControlNumber(13), PClonable>
+        <PropertyOption(ControlText:="Download stories: user", Category:=CAT_DOWN), PXML, ControlNumber(13), PClonable>
         Friend ReadOnly Property DownloadStoriesUser As PropertyValue
         <PXML> Private ReadOnly Property DownloadStoriesUser_Def As PropertyValue
-        <PropertyOption(ControlText:="Download tagged", ControlToolTip:="Download tagged posts", Category:=CAT_DOWN), PXML, ControlNumber(14), PClonable>
+        <PropertyOption(ControlText:="Download tagged posts", Category:=CAT_DOWN), PXML, ControlNumber(14), PClonable>
         Friend ReadOnly Property DownloadTagged As PropertyValue
         <PXML> Private ReadOnly Property DownloadTagged_Def As PropertyValue
 #End Region
@@ -425,10 +440,17 @@ Namespace API.Instagram
             SleepTimerOnPostsLimitProvider = New TimersChecker(10000)
 
             GetTimeline = New PropertyValue(True)
+            GetTimeline_VideoPic = New PropertyValue(True)
             GetReels = New PropertyValue(False)
+            GetReels_VideoPic = New PropertyValue(True)
             GetStories = New PropertyValue(False)
+            GetStories_VideoPic = New PropertyValue(True)
             GetStoriesUser = New PropertyValue(False)
+            GetStoriesUser_VideoPic = New PropertyValue(True)
             GetTagged = New PropertyValue(False)
+            GetTagged_VideoPic = New PropertyValue(True)
+            GetSavedPosts_VideoPic = New PropertyValue(True)
+            PutImageVideoFolder = New PropertyValue(False)
             TaggedNotifyLimit = New PropertyValue(200)
             TaggedNotifyLimitProvider = New TaggedNotifyLimitChecker
 

SCrawler/API/Instagram/UserData.GQL.vb

@@ -194,7 +194,7 @@ Namespace API.Instagram
                             With j({"data", "xdt_api__v1__feed__reels_media__connection", "edges"})
                                 If .ListExists Then
                                     ProgressPre.ChangeMax(.Count)
-                                    For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token) : Next
+                                    For Each n As EContainer In .Self : GetStoriesData_ParseSingleHighlight(n("node"), i, False, Token, Sections.Stories) : Next
                                 End If
                             End With
                         End If
@@ -217,7 +217,7 @@ Namespace API.Instagram
                 Using j As EContainer = JsonDocument.Parse(r)
                     If j.ListExists Then
                         Dim i% = -1
-                        GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token)
+                        GetStoriesData_ParseSingleHighlight(j.ItemF({"data", "xdt_api__v1__feed__reels_media", "reels_media", 0}), i, True, Token, Sections.UserStories)
                     End If
                 End Using
             End If

SCrawler/API/Instagram/UserData.vb

@@ -26,10 +26,16 @@ Namespace API.Instagram
         Private Const Name_LastCursor As String = "LastCursor"
         Private Const Name_FirstLoadingDone As String = "FirstLoadingDone"
         Private Const Name_GetTimeline As String = "GetTimeline"
+        Private Const Name_GetTimeline_VideoPic As String = "GetTimeline_VideoPic"
         Private Const Name_GetReels As String = "GetReels"
+        Private Const Name_GetReels_VideoPic As String = "GetReels_VideoPic"
         Private Const Name_GetStories As String = "GetStories"
+        Private Const Name_GetStories_VideoPic As String = "GetStories_VideoPic"
         Private Const Name_GetStoriesUser As String = "GetStoriesUser"
+        Private Const Name_GetStoriesUser_VideoPic As String = "GetStoriesUser_VideoPic"
         Private Const Name_GetTagged As String = "GetTaggedData"
+        Private Const Name_GetTagged_VideoPic As String = "GetTaggedData_VideoPic"
+        Private Const Name_PutImageVideoFolder As String = "PutImageVideoFolder"
         Private Const Name_TaggedChecked As String = "TaggedChecked"
         Private Const Name_NameTrue As String = "NameTrue"
 #End Region
@@ -79,10 +85,32 @@ Namespace API.Instagram
         Private LastCursor As String = String.Empty
         Private FirstLoadingDone As Boolean = False
         Friend Property GetTimeline As Boolean = True
+        Friend Property GetTimeline_VideoPic As Boolean = True
         Friend Property GetReels As Boolean = False
+        Friend Property GetReels_VideoPic As Boolean = True
         Friend Property GetStories As Boolean
+        Friend Property GetStories_VideoPic As Boolean = True
         Friend Property GetStoriesUser As Boolean
+        Friend Property GetStoriesUser_VideoPic As Boolean = True
         Friend Property GetTaggedData As Boolean
+        Friend Property GetTaggedData_VideoPic As Boolean = True
+        Friend Property PutImageVideoFolder As Boolean = False
+        Private Function ExtractImageFrom(ByVal Section As Sections) As Boolean
+            Select Case Section
+                Case Sections.Timeline : Return GetTimeline_VideoPic
+                Case Sections.Reels : Return GetReels_VideoPic
+                Case Sections.Tagged : Return GetTaggedData_VideoPic
+                Case Sections.Stories : Return GetStories_VideoPic
+                Case Sections.UserStories : Return GetStoriesUser_VideoPic
+                Case Sections.SavedPosts
+                    Try
+                        If Not HOST Is Nothing AndAlso HOST.Key = InstagramSiteKey Then Return MySiteSettings.GetSavedPosts_VideoPic.Value
+                    Catch
+                    End Try
+                    Return True
+                Case Else : Return True
+            End Select
+        End Function
         Protected _NameTrue As String = String.Empty
         Friend ReadOnly Property NameTrue As String
             Get
@@ -98,20 +126,32 @@ Namespace API.Instagram
                     LastCursor = .Value(Name_LastCursor)
                     FirstLoadingDone = .Value(Name_FirstLoadingDone).FromXML(Of Boolean)(False)
                     GetTimeline = .Value(Name_GetTimeline).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline.Value))
-                    GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(MySiteSettings.GetReels.Value)
+                    GetTimeline_VideoPic = .Value(Name_GetTimeline_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTimeline_VideoPic.Value))
+                    GetReels = .Value(Name_GetReels).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels.Value))
+                    GetReels_VideoPic = .Value(Name_GetReels_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetReels_VideoPic.Value))
                     GetStories = .Value(Name_GetStories).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories.Value))
-                    GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(MySiteSettings.GetStoriesUser.Value)
+                    GetStories_VideoPic = .Value(Name_GetStories_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStories_VideoPic.Value))
+                    GetStoriesUser = .Value(Name_GetStoriesUser).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser.Value))
+                    GetStoriesUser_VideoPic = .Value(Name_GetStoriesUser_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetStoriesUser_VideoPic.Value))
+                    PutImageVideoFolder = .Value(Name_PutImageVideoFolder).FromXML(Of Boolean)(CBool(MySiteSettings.PutImageVideoFolder.Value))
                     GetTaggedData = .Value(Name_GetTagged).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged.Value))
+                    GetTaggedData_VideoPic = .Value(Name_GetTagged_VideoPic).FromXML(Of Boolean)(CBool(MySiteSettings.GetTagged_VideoPic.Value))
                     TaggedChecked = .Value(Name_TaggedChecked).FromXML(Of Boolean)(False)
                     _NameTrue = .Value(Name_NameTrue)
                 Else
                     .Add(Name_LastCursor, LastCursor)
                     .Add(Name_FirstLoadingDone, FirstLoadingDone.BoolToInteger)
                     .Add(Name_GetTimeline, GetTimeline.BoolToInteger)
+                    .Add(Name_GetTimeline_VideoPic, GetTimeline_VideoPic.BoolToInteger)
                     .Add(Name_GetReels, GetReels.BoolToInteger)
+                    .Add(Name_GetReels_VideoPic, GetReels_VideoPic.BoolToInteger)
                     .Add(Name_GetStories, GetStories.BoolToInteger)
+                    .Add(Name_GetStories_VideoPic, GetStories_VideoPic.BoolToInteger)
                     .Add(Name_GetStoriesUser, GetStoriesUser.BoolToInteger)
+                    .Add(Name_GetStoriesUser_VideoPic, GetStoriesUser_VideoPic.BoolToInteger)
                     .Add(Name_GetTagged, GetTaggedData.BoolToInteger)
+                    .Add(Name_GetTagged_VideoPic, GetTaggedData_VideoPic.BoolToInteger)
+                    .Add(Name_PutImageVideoFolder, PutImageVideoFolder.BoolToInteger)
                     .Add(Name_TaggedChecked, TaggedChecked.BoolToInteger)
                     .Add(Name_NameTrue, _NameTrue)
                 End If
@@ -130,6 +170,14 @@ Namespace API.Instagram
                     GetStories = .GetStories
                     GetStoriesUser = .GetStoriesUser
                     GetTaggedData = .GetTagged
+
+                    GetTimeline_VideoPic = .GetTimeline_VideoPic
+                    GetReels_VideoPic = .GetReels_VideoPic
+                    GetStories_VideoPic = .GetStories_VideoPic
+                    GetStoriesUser_VideoPic = .GetStoriesUser_VideoPic
+                    GetTaggedData_VideoPic = .GetTagged_VideoPic
+
+                    PutImageVideoFolder = .PutImageVideoFolder
                 End With
             End If
         End Sub
@@ -809,7 +857,7 @@ NextPageBlock:
                                             With j("items")
                                                 For Each jj In .Self
                                                     before = _TempMediaList.Count
-                                                    ObtainMedia(jj, PostsToReparse(i).ID, specFolder)
+                                                    ObtainMedia(jj, PostsToReparse(i).ID, specFolder,,,,,,, IIf(IsTagged, Sections.Tagged, Sections.Timeline))
                                                     If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
                                                     If _Limit > 0 And _TotalPostsParsed >= _Limit Then Throw New ExitException
                                                 Next
@@ -911,7 +959,7 @@ NextPageBlock:
                                     End Select
                                 End If
                                 before = _TempMediaList.Count
-                                ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts)
+                                ObtainMedia(.Self, PostIDKV.ID, SpecFolder, PostDate,, PostOriginUrl, State, Attempts,, Section)
                                 If Not before = _TempMediaList.Count Then _TotalPostsParsed += 1
                                 If _Limit > 0 And _TotalPostsParsed >= _Limit Then Return False
                             End If
@@ -950,6 +998,7 @@ NextPageBlock:
         Protected ObtainMedia_SizeFuncVid As Func(Of EContainer, Sizes) = Nothing
         Protected ObtainMedia_SizeFuncPic As Func(Of EContainer, Sizes) = Nothing
         Protected ObtainMedia_AllowAbstract As Boolean = False
+        Private Const ObtainMedia_NoSection As Integer = -10
         Protected Sub ObtainMedia_SetReelsFunc()
             ObtainMedia_SizeFuncPic = Function(ByVal ss As EContainer) As Sizes
                                           If ss.Value("url").IsEmptyString Then
@@ -971,7 +1020,8 @@ NextPageBlock:
                                   Optional ByVal DateObj As String = Nothing, Optional ByVal InitialType As Integer = -1,
                                   Optional ByVal PostOriginUrl As String = Nothing,
                                   Optional ByVal State As UStates = UStates.Unknown, Optional ByVal Attempts As Integer = 0,
-                                  Optional ByVal TryExtractImage As Boolean = False)
+                                  Optional ByVal TryExtractImage As Boolean = False,
+                                  Optional ByVal Section As Sections = ObtainMedia_NoSection)
             Try
                 Dim maxSize As Func(Of EContainer, Integer) = Function(ByVal _ss As EContainer) As Integer
                                                                   Dim w% = AConvert(Of Integer)(_ss.Value("width"), 0)
@@ -1018,6 +1068,12 @@ NextPageBlock:
                     If TryExtractImage Then
                         t = 1
                         abstractDecision = True
+                        If Not SpecialFolder.IsEmptyString AndAlso PutImageVideoFolder Then
+                            Dim endsAbs As Boolean = SpecialFolder.EndsWith("*")
+                            If endsAbs Then SpecialFolder = SpecialFolder.TrimEnd("*")
+                            If Not SpecialFolder.IsEmptyString Then SpecialFolder = $"{SpecialFolder.TrimEnd("\")}\{VideoFolderName}{IIf(Not endsAbs, "*", String.Empty)}"
+                            If endsAbs Then SpecialFolder &= "*"
+                        End If
                     ElseIf t = -1 And InitialType = 8 And ObtainMedia_AllowAbstract Then
                         If n.Contains(vid) Then
                             t = 2
@@ -1064,7 +1120,8 @@ NextPageBlock:
                                         End If
                                     End With
                                 End If
-                                If Not TryExtractImage Then ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True)
+                                If Not TryExtractImage And Not Section = ObtainMedia_NoSection And ExtractImageFrom(Section) Then _
+                                   ObtainMedia(n, PostID, SpecialFolder, DateObj, InitialType, PostOriginUrl, State, Attempts, True, Section)
                             Case 8 'gallery
                                 DateObj = mDate(n)
                                 With n("carousel_media").XmlIfNothing
@@ -1165,6 +1222,7 @@ NextPageBlock:
             Dim qStr$, r$
             Dim i% = -1
             Dim jj As EContainer
+            Dim section As Sections = IIf(GetUserStory, Sections.UserStories, Sections.Stories)
             ThrowAny(Token)
             If StoriesList.ListExists Or GetUserStory Then
                 If Not GetUserStory Then tmpList = StoriesList.Take(5)
@@ -1181,7 +1239,7 @@ NextPageBlock:
                         Using j As EContainer = JsonDocument.Parse(r).XmlIfNothing
                             If j.Contains("reels") Then
                                 ProgressPre.ChangeMax(j("reels").Count)
-                                For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token) : Next
+                                For Each jj In j("reels") : GetStoriesData_ParseSingleHighlight(jj, i, GetUserStory, Token, section) : Next
                             End If
                         End Using
                     End If
@@ -1189,7 +1247,8 @@ NextPageBlock:
                 End If
             End If
         End Sub
-        Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean, ByVal Token As CancellationToken)
+        Private Sub GetStoriesData_ParseSingleHighlight(ByVal Node As EContainer, ByRef Index As Integer, ByVal GetUserStory As Boolean,
+                                                        ByVal Token As CancellationToken, Optional ByVal Section As Sections = Sections.Stories)
             If Not Node Is Nothing Then
                 With Node
                     ProgressPre.Perform()
@@ -1210,7 +1269,7 @@ NextPageBlock:
                                 pid = storyID & s.Value("id")
                                 If Not _TempPostsList.Contains(pid) Then
                                     ThrowAny(Token)
-                                    ObtainMedia(s, pid, sFolder)
+                                    ObtainMedia(s, pid, sFolder,,,,,,, Section)
                                     _TempPostsList.Add(pid)
                                 End If
                             Next

SCrawler/API/OnlyFans/Declarations.vb

@@ -11,6 +11,11 @@ Namespace API.OnlyFans
     Friend Module Declarations
         Friend ReadOnly DateProvider As New ADateTime("O")
         Friend ReadOnly RegExPostID As RParams = RParams.DM("(?<=onlyfans\.com/)(\d+)", 0, EDP.ReturnValue)
+        Friend ReadOnly FilesSources As New List(Of Object()) From {
+            {{"source", "source"}},
+            {{"files", "source", "url"}},
+            {{"files", "full", "url"}}
+        }
         Friend Property Rules As DynamicRulesEnv
     End Module
 End Namespace

SCrawler/API/OnlyFans/UserData.vb

@@ -394,6 +394,14 @@ Namespace API.OnlyFans
             Loop While Not _complete
         End Sub
 #End Region
+        Private Function GetMediaURL(ByVal m As EContainer) As String
+            Dim v$
+            For Each node As Object() In FilesSources
+                v = If(m.ItemF(node)?.Value, String.Empty)
+                If Not v.IsEmptyString Then Return v
+            Next
+            Return String.Empty
+        End Function
         Private Function TryCreateMedia(ByVal n As EContainer, ByVal PostID As String, Optional ByVal PostDate As String = Nothing,
                                         Optional ByRef Result As Boolean = False, Optional ByVal IsHL As Boolean = False,
                                         Optional ByVal SpecFolder As String = Nothing, Optional ByVal PostUserID As String = Nothing,
@@ -405,11 +413,14 @@ Namespace API.OnlyFans
             With n("media")
                 If .ListExists Then
                     For Each m In .Self
-                        If IsHL Then
+                        postUrl = GetMediaURL(m)
-                            postUrl = m.Value({"files", "source"}, "url")
+                        'If IsHL Then
-                        Else
+                        '    'postUrl = m.Value({"files", "source"}, "url")
-                            postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
+                        '    postUrl = GetMediaURL(m)
-                        End If
+                        'Else
+                        '    'postUrl = m.Value({"source"}, "source").IfNullOrEmpty(m.Value("full"))
+                        '    postUrl = GetMediaURL(m)
+                        'End If
                         postUrlBase = String.Empty
                         Select Case m.Value("type")
                             Case "photo" : t = UTypes.Picture : ext = "jpg"

SCrawler/API/Twitter/UserData.vb

@@ -90,6 +90,7 @@ Namespace API.Twitter
                     GifsPrefix = .GifsPrefix
                     UseMD5Comparison = .UseMD5Comparison
                     RemoveExistingDuplicates = .RemoveExistingDuplicates
+                    If RemoveExistingDuplicates Then StartMD5Checked = False
                     DownloadModel = DownloadModels.Undefined
                     DownloadModelForceApply = .DownloadModelForceApply
                     MediaModelAllowNonUserTweets = .MediaModelAllowNonUserTweets

SCrawler/API/YouTube/UserData.vb

@@ -203,7 +203,7 @@ Namespace API.YouTube
                     If IsMusic Or DownloadYTVideos Then
                         maxDate = Nothing
                         LastDownloadDateVideos = nDate(LastDownloadDateVideos)
-                        url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}"
+                        url = $"https://{IIf(IsMusic, "music", "www")}.youtube.com/{IIf(IsMusic Or IsChannelUser, $"{YouTubeFunctions.UserChannelOption}/", "@")}{ID}/videos"
                         container = YouTubeFunctions.Parse(url, YTUseCookies, Token, pr, __getMinDate(LastDownloadDateVideos), __maxDate,, True)
                         applySpecFolder.Invoke(IIf(IsMusic, String.Empty, "Videos"), False)
                         If fillList.Invoke(LastDownloadDateVideos, False) Then LastDownloadDateVideos = If(maxDate, Now)

SCrawler/Download/Feed/DownloadFeedForm.vb

@@ -550,7 +550,7 @@ Namespace DownloadObjects
         End Sub
         Private Function MoveCopyFiles(ByVal IsInternal As Boolean, ByVal Sender As Object, ByVal MCTOptions As FeedMoveCopyTo,
                                        ByVal FeedMediaData As FeedMedia, Optional ByVal GetChecked As Boolean = True) As Boolean
-            Const MsgTitle$ = "Copy/Move checked files"
+            Dim MsgTitle$ = "Copy/Move checked files"
             Try
                 Dim isCopy As Boolean = Not Sender Is Nothing AndAlso (Sender Is BTT_COPY_TO OrElse Sender Is BTT_COPY_SPEC_TO)
                 Dim moveOptions As FeedMoveCopyTo = Nothing
@@ -591,7 +591,18 @@ Namespace DownloadObjects
                     data = {FeedMediaData.Media}
                     data_files = {FeedMediaData.File}
                 End If
+
+                MsgTitle = $"{IIf(isCopy, "Copy", "Move")} {IIf(Not FeedMediaData Is Nothing Or GetChecked, "checked", "ALL")} files"
+
                 If data.ListExists Then
+
+                    If (FeedMediaData Is Nothing And Not GetChecked And Not isCopy) AndAlso
+                       MsgBoxE({$"YOU ARE TRYING TO MOVE ALL FEED/SESSION DATA.{vbCr}EVERY FILE WILL BE MOVED, NOT JUST THE SELECTED ONES.", MsgTitle},
+                               vbExclamation,,, {"Process", "Cancel"}) = 1 Then
+                        ShowOperationCanceledMsg(MsgTitle)
+                        Return False
+                    End If
+
                     If MCTOptions.Destination.IsEmptyString Then
                         Using f As New FeedCopyToForm(data_files, isCopy)
                             f.ShowDialog()

SCrawler/Download/Feed/FeedMedia.Designer.vb

@@ -27,7 +27,6 @@ Namespace DownloadObjects
             Dim TP_LBL As System.Windows.Forms.TableLayoutPanel
             Dim CONTEXT_SEP_3 As System.Windows.Forms.ToolStripSeparator
             Dim CONTEXT_SEP_4 As System.Windows.Forms.ToolStripSeparator
-            Me.CONTEXT_SEP_2 = New System.Windows.Forms.ToolStripSeparator()
             Me.CH_CHECKED = New System.Windows.Forms.CheckBox()
             Me.LBL_INFO = New System.Windows.Forms.Label()
             Me.CONTEXT_DATA = New System.Windows.Forms.ContextMenuStrip(Me.components)
@@ -37,6 +36,7 @@ Namespace DownloadObjects
             Me.BTT_CONTEXT_OPEN_USER = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_CONTEXT_OPEN_USER_URL = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_CONTEXT_OPEN_USER_POST = New System.Windows.Forms.ToolStripMenuItem()
+            Me.CONTEXT_SEP_2 = New System.Windows.Forms.ToolStripSeparator()
             Me.BTT_COPY_TO = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_MOVE_TO = New System.Windows.Forms.ToolStripMenuItem()
             Me.BTT_FEED_ADD_FAV = New System.Windows.Forms.ToolStripMenuItem()
@@ -52,6 +52,7 @@ Namespace DownloadObjects
             Me.ICON_SITE = New System.Windows.Forms.PictureBox()
             Me.TP_MAIN = New System.Windows.Forms.TableLayoutPanel()
             Me.LBL_TITLE = New System.Windows.Forms.Label()
+            Me.BTT_CONTEXT_OPEN_FILE_FOLDER = New System.Windows.Forms.ToolStripMenuItem()
             CONTEXT_SEP_1 = New System.Windows.Forms.ToolStripSeparator()
             TP_LBL = New System.Windows.Forms.TableLayoutPanel()
             CONTEXT_SEP_3 = New System.Windows.Forms.ToolStripSeparator()
@@ -67,11 +68,6 @@ Namespace DownloadObjects
             CONTEXT_SEP_1.Name = "CONTEXT_SEP_1"
             CONTEXT_SEP_1.Size = New System.Drawing.Size(302, 6)
             '
-            'CONTEXT_SEP_2
-            '
-            Me.CONTEXT_SEP_2.Name = "CONTEXT_SEP_2"
-            Me.CONTEXT_SEP_2.Size = New System.Drawing.Size(302, 6)
-            '
             'TP_LBL
             '
             TP_LBL.ColumnCount = 3
@@ -114,9 +110,9 @@ Namespace DownloadObjects
             '
             'CONTEXT_DATA
             '
-            Me.CONTEXT_DATA.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_CONTEXT_DOWN, Me.CONTEXT_SEP_0, Me.BTT_CONTEXT_OPEN_MEDIA, Me.BTT_CONTEXT_OPEN_USER, CONTEXT_SEP_1, Me.BTT_CONTEXT_OPEN_USER_URL, Me.BTT_CONTEXT_OPEN_USER_POST, Me.CONTEXT_SEP_2, Me.BTT_COPY_TO, Me.BTT_MOVE_TO, CONTEXT_SEP_3, Me.BTT_FEED_ADD_FAV, Me.BTT_FEED_ADD_FAV_REMOVE, Me.BTT_FEED_ADD_SPEC, Me.BTT_FEED_ADD_SPEC_REMOVE, Me.BTT_FEED_REMOVE_FAV, Me.BTT_FEED_REMOVE_SPEC, CONTEXT_SEP_4, Me.BTT_CONTEXT_FIND_USER, Me.BTT_CONTEXT_INFO, Me.CONTEXT_SEP_5, Me.BTT_CONTEXT_DELETE})
+            Me.CONTEXT_DATA.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_CONTEXT_DOWN, Me.CONTEXT_SEP_0, Me.BTT_CONTEXT_OPEN_MEDIA, Me.BTT_CONTEXT_OPEN_USER, Me.BTT_CONTEXT_OPEN_FILE_FOLDER, CONTEXT_SEP_1, Me.BTT_CONTEXT_OPEN_USER_URL, Me.BTT_CONTEXT_OPEN_USER_POST, Me.CONTEXT_SEP_2, Me.BTT_COPY_TO, Me.BTT_MOVE_TO, CONTEXT_SEP_3, Me.BTT_FEED_ADD_FAV, Me.BTT_FEED_ADD_FAV_REMOVE, Me.BTT_FEED_ADD_SPEC, Me.BTT_FEED_ADD_SPEC_REMOVE, Me.BTT_FEED_REMOVE_FAV, Me.BTT_FEED_REMOVE_SPEC, CONTEXT_SEP_4, Me.BTT_CONTEXT_FIND_USER, Me.BTT_CONTEXT_INFO, Me.CONTEXT_SEP_5, Me.BTT_CONTEXT_DELETE})
             Me.CONTEXT_DATA.Name = "CONTEXT_PIC"
-            Me.CONTEXT_DATA.Size = New System.Drawing.Size(306, 414)
+            Me.CONTEXT_DATA.Size = New System.Drawing.Size(306, 436)
             '
             'BTT_CONTEXT_DOWN
             '
@@ -160,6 +156,11 @@ Namespace DownloadObjects
             Me.BTT_CONTEXT_OPEN_USER_POST.Size = New System.Drawing.Size(305, 22)
             Me.BTT_CONTEXT_OPEN_USER_POST.Text = "Open post"
             '
+            'CONTEXT_SEP_2
+            '
+            Me.CONTEXT_SEP_2.Name = "CONTEXT_SEP_2"
+            Me.CONTEXT_SEP_2.Size = New System.Drawing.Size(302, 6)
+            '
             'BTT_COPY_TO
             '
             Me.BTT_COPY_TO.Image = Global.SCrawler.My.Resources.Resources.PastePic_32
@@ -289,6 +290,13 @@ Namespace DownloadObjects
             Me.LBL_TITLE.Size = New System.Drawing.Size(140, 25)
             Me.LBL_TITLE.TabIndex = 1
             '
+            'BTT_CONTEXT_OPEN_FILE_FOLDER
+            '
+            Me.BTT_CONTEXT_OPEN_FILE_FOLDER.Image = Global.SCrawler.My.Resources.Resources.FolderPic_32
+            Me.BTT_CONTEXT_OPEN_FILE_FOLDER.Name = "BTT_CONTEXT_OPEN_FILE_FOLDER"
+            Me.BTT_CONTEXT_OPEN_FILE_FOLDER.Size = New System.Drawing.Size(305, 22)
+            Me.BTT_CONTEXT_OPEN_FILE_FOLDER.Text = "Open file folder"
+            '
             'FeedMedia
             '
             Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
@@ -335,5 +343,6 @@ Namespace DownloadObjects
         Private WithEvents BTT_MOVE_TO As ToolStripMenuItem
         Private WithEvents CONTEXT_SEP_5 As ToolStripSeparator
         Private WithEvents CONTEXT_SEP_2 As ToolStripSeparator
+        Private WithEvents BTT_CONTEXT_OPEN_FILE_FOLDER As ToolStripMenuItem
     End Class
 End Namespace

SCrawler/Download/Feed/FeedMedia.vb

@@ -178,6 +178,7 @@ Namespace DownloadObjects
                     BTT_CONTEXT_DOWN.Visible = True
                     CONTEXT_SEP_0.Visible = True
                     BTT_CONTEXT_OPEN_USER.Visible = False
+                    BTT_CONTEXT_OPEN_FILE_FOLDER.Visible = False
                     CONTEXT_SEP_5.Visible = False
                     BTT_CONTEXT_DELETE.Visible = False
 
@@ -416,8 +417,12 @@ Namespace DownloadObjects
         End Sub
 #End Region
 #Region "Open media, folder"
-        Private Sub BTT_CONTEXT_OPEN_MEDIA_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_OPEN_MEDIA.Click
+        Private Sub BTT_CONTEXT_OPEN_MEDIA_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_OPEN_MEDIA.Click, BTT_CONTEXT_OPEN_FILE_FOLDER.Click
-            File.Open()
+            If Not sender Is Nothing AndAlso sender Is BTT_CONTEXT_OPEN_FILE_FOLDER Then
+                GlobalOpenPath(File)
+            Else
+                File.Open()
+            End If
         End Sub
         Private Sub BTT_CONTEXT_OPEN_USER_Click(sender As Object, e As EventArgs) Handles BTT_CONTEXT_OPEN_USER.Click
             If Not UserKey.IsEmptyString Then

SCrawler/MainFrame.vb

@@ -247,7 +247,7 @@ CloseResume:
                 BTT_DOWN_AUTOMATION.PerformClick()
             ElseIf e.Alt And e.KeyCode = Keys.P Then
                 BTT_PR_INFO.PerformClick()
-            ElseIf e.Alt And e.KeyCode = Keys.F Then
+            ElseIf (e.Alt And (e.KeyCode = Keys.F Or e.KeyCode = Keys.U)) Or (e.Control And e.KeyCode = Keys.U) Then
                 MySearch.FormShow()
             Else
                 b = False

SCrawler/My Project/AssemblyInfo.vb

@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
 ' by using the '*' as shown below:
 ' <Assembly: AssemblyVersion("1.0.*")>
 
-<Assembly: AssemblyVersion("2024.8.1.0")>
+<Assembly: AssemblyVersion("2024.9.2.0")>
-<Assembly: AssemblyFileVersion("2024.8.1.0")>
+<Assembly: AssemblyFileVersion("2024.9.2.0")>
 <Assembly: NeutralResourcesLanguage("en")>

SCrawler/UserImage.vb

@@ -13,6 +13,25 @@ Friend Class UserImage : Inherits ImageRenderer
     Friend Const ImagePostfix_Small As String = "_Small"
     Private _LargeAddress As SFile
     Private _SmallAddress As SFile
+    Private _ForceSaveOrig As Boolean = False
+    Friend Shared Function NewUserPicture(ByVal ImageOrig As SFile, ByVal Destination As SFile,
+                                          Optional ByVal Save As Boolean = True, Optional ByVal GetInstance As Boolean = False) As UserImage
+        Dim uImg As New UserImage(ImageOrig, Destination)
+        With uImg
+            ._ForceSaveOrig = ImageOrig.Extension.IsEmptyString OrElse ImageOrig.Extension.ToLower = "gif" OrElse Not {"jpg", "jpeg", "png"}.Contains(ImageOrig.Extension.ToLower)
+            If Not ._ForceSaveOrig Then
+                If .Address.Exists AndAlso Not .Address.Delete(SFO.File,, EDP.ReturnValue) Then ._ForceSaveOrig = True
+                If Not ._ForceSaveOrig AndAlso Not ImageOrig.Copy(.Address) Then ._ForceSaveOrig = True
+            End If
+            If Not ._ForceSaveOrig Then
+                ._SmallAddress.Extension = .Address.Extension
+                ._LargeAddress.Extension = .Address.Extension
+            End If
+            If Save Then .Save()
+        End With
+        If Not GetInstance Then uImg.Dispose() : uImg = Nothing
+        Return uImg
+    End Function
     Friend Sub New(ByVal _ImgOriginal As SFile, ByVal Destination As SFile, Optional ByVal GenerateLargeSmallPictures As Boolean = True)
         MyBase.New(_ImgOriginal)
         Dim f As SFile = Destination
@@ -71,7 +90,7 @@ Friend Class UserImage : Inherits ImageRenderer
         End With
     End Function
     Public Overrides Sub Save()
-        MyBase.Save()
+        If _ForceSaveOrig Then MyBase.Save()
         Small.Save(_SmallAddress)
         Large.Save(_LargeAddress)
     End Sub

Tools/ArchiveSCrawlerUsersDataFiles.bat

@@ -4,4 +4,8 @@ REM Replace 'd:\Downloads\SocialNetworks\' with the path to your SCrawler data f
 REM THIS SCRIPT IS NOT SUITABLE FOR 7ZIP OR OTHER ARCHIVING PROGRAMS.
 REM But I believe 7Zip also has CLI commands
 
-"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "d:\Downloads\SocialNetworks\SCrawlerBackup.rar" "d:\Downloads\SocialNetworks\"
+REM This line archives SCrawler settings files.
+"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl "D:\MyPrograms\SCrawler\Backup\Settings.rar" "D:\MyPrograms\SCrawler\Settings\"
+
+REM This line archives SCrawler users' settings files.
+"C:\Program Files\WinRAR\WinRAR.exe" a -r -ep1 -o+ -ag_YYYYMMDD_HHMMSS -m5 -tl -n*.txt -n*.xml "D:\MyPrograms\SCrawler\Backup\SCrawlerBackup.rar" "D:\MyPrograms\SCrawler\Data\"