mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-15 00:02:17 +00:00
Compare commits
4 Commits
2023.1.2.0
...
2023.1.24.
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fc226d549a | ||
|
|
602771d982 | ||
|
|
3e472b4f2b | ||
|
|
30c3fe3b68 |
@@ -8,14 +8,16 @@ I welcome requests! Follow these steps to contribute:
|
||||
1. If you have a code change suggestion, you can post a replacement code block. I also accept pull requests.
|
||||
|
||||
# How to build from source
|
||||
1. Delete the "PersonalUtilities" project from the solution.
|
||||
1. Delete the "PersonalUtilities.Notifications" project from the solution.
|
||||
1. Delete the ```PersonalUtilities``` project from the solution.
|
||||
1. Delete the ```PersonalUtilities.Notifications``` project from the solution.
|
||||
1. Delete the ```cURL``` folder from the solution.
|
||||
1. Delete the ```ffmpeg.exe``` from the solution.
|
||||
1. The following libraries must be added to project references with the '**Copy to output folder**' option:
|
||||
- ```PersonalUtilities.dll```
|
||||
- ```PersonalUtilities.Notifications.dll```
|
||||
- ```Microsoft.Toolkit.Uwp.Notifications.dll```
|
||||
- ```System.ValueTuple.dll```
|
||||
1. Import PersonalUtilities.Functions for the whole project.
|
||||
1. Import ```PersonalUtilities.Functions``` for the whole project.
|
||||
|
||||
**Always use the correct libraries. You must download libraries from the same release date as the code commit date.**
|
||||
|
||||
|
||||
17
Changelog.md
17
Changelog.md
@@ -1,3 +1,20 @@
|
||||
# 2023.1.24.1
|
||||
|
||||
*2023-01-24*
|
||||
|
||||
- Added
|
||||
- Icon for standalone downloader
|
||||
- Fixed
|
||||
- (Issue #100) some Imgur albums won't download
|
||||
|
||||
# 2023.1.24.0
|
||||
|
||||
*2023-01-24*
|
||||
|
||||
- Fixed
|
||||
- (Issue #100) Imgur albums not downloading
|
||||
- When deleting a collection with the 'ban' option, users in the collection are not banned
|
||||
|
||||
# 2023.1.2.0
|
||||
|
||||
*2023-01-02*
|
||||
|
||||
@@ -6,6 +6,7 @@ You can support the program by:
|
||||
- :repeat: make a post about my program on your profile (Reddit, Twitter, Instagram and any other social networks)
|
||||
- :speech_balloon: tell your friends about the program
|
||||
- :heart: like the program on this site: https://alternativeto.net/software/scrawler/about/
|
||||
- :heart: like the program on this site: https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml
|
||||
- suggest my program as an alternative ([on this site](https://alternativeto.net/software/scrawler/about/)) to any program you have used before
|
||||
|
||||
I would be very grateful for any support! :blush:
|
||||
|
||||
26
README.md
26
README.md
@@ -1,6 +1,4 @@
|
||||
# :rainbow_flag: Social networks crawler :rainbow_flag: :christmas_tree:
|
||||
|
||||
# :christmas_tree: Happy new year :christmas_tree:
|
||||
# :rainbow_flag: Social networks crawler :rainbow_flag:
|
||||
|
||||
[](https://github.com/AAndyProgram/SCrawler/releases/latest)
|
||||
[](https://github.com/AAndyProgram/SCrawler/blob/main/LICENSE)
|
||||
@@ -39,18 +37,18 @@ Do you like this program? Consider adding to my coffee fund by making a donation
|
||||
- Download [saved Reddit, Twitter and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
|
||||
- Add users from parsed channel
|
||||
- **Advanced user management**
|
||||
- **Automation** (downloading data automatically every ```X``` minutes)
|
||||
- **Feed** (feed of downloaded media files)
|
||||
- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every ```X``` minutes)
|
||||
- **Feed** ([feed](https://github.com/AAndyProgram/SCrawler/wiki#feed) of downloaded media files)
|
||||
- Labeling users
|
||||
- Create download groups
|
||||
- Create [download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
|
||||
- Adding users to favorites and temporary
|
||||
- Filter exists users by label or group
|
||||
- [Filter exists users](https://github.com/AAndyProgram/SCrawler/wiki#view) by label or group
|
||||
- Selection of media types you want to download (images only, videos only, both)
|
||||
- Download a special video, image or gallery
|
||||
- Making collections (grouping users into collections)
|
||||
- [Download a special video](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video), image or gallery
|
||||
- Making [collections](https://github.com/AAndyProgram/SCrawler/wiki#collections) (grouping users into collections)
|
||||
- Specifying a user folder (for downloading data to another location)
|
||||
- Changing user icons
|
||||
- Changing view modes
|
||||
- Changing [view modes](https://github.com/AAndyProgram/SCrawler/wiki#view)
|
||||
- ...and many others...
|
||||
|
||||
# Supported sites
|
||||
@@ -76,11 +74,11 @@ First, the program downloads the full profile. After the program downloads only
|
||||
|
||||
## Reddit
|
||||
|
||||
The program parses all user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
|
||||
The program parses user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
|
||||
|
||||
## Other sites
|
||||
|
||||
The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
|
||||
The program parses user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
|
||||
|
||||
## How to request a new site
|
||||
|
||||
@@ -122,7 +120,7 @@ Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
|
||||
|
||||
# Installation
|
||||
|
||||
**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy.** :blush:
|
||||
**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder and enjoy.** :blush:
|
||||
|
||||
**Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
|
||||
|
||||
@@ -140,7 +138,7 @@ Read about how to make plugin [here](https://github.com/AAndyProgram/SCrawler/wi
|
||||
|
||||
# How to support
|
||||
|
||||
Read more about how to support the program [here](HowToSupport.md).
|
||||
Read about how to support the program [here](HowToSupport.md).
|
||||
|
||||
# Settings and usage
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ Imports PersonalUtilities.Tools.Web.Documents.JSON
|
||||
Namespace API.Imgur
|
||||
Namespace Declarations
|
||||
Friend Module Imgur_Declarations
|
||||
Friend ReadOnly PostRegex As RParams = RParams.DMS("/([\w\d]+?)(|\.[\w]{0,4})\Z", 1)
|
||||
Friend ReadOnly PostRegex As RParams = RParams.DMS("/([^/]+?)(|#.*?|\.[\w]{0,4})(|\?.*?)\Z", 1)
|
||||
End Module
|
||||
End Namespace
|
||||
Friend NotInheritable Class Envir
|
||||
@@ -70,11 +70,12 @@ Namespace API.Imgur
|
||||
Friend Shared Function GetVideoInfo(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia)
|
||||
Try
|
||||
If Not URL.IsEmptyString AndAlso URL.ToLower.Contains("imgur") AndAlso Not Settings.ImgurClientID.IsEmptyString Then
|
||||
Dim img$ = GetImage(URL, EDP.ReturnValue)
|
||||
If Not img.IsEmptyString Then
|
||||
Return {New UserMedia(img)}
|
||||
Dim imgList As List(Of String) = GetGallery(URL, EDP.ReturnValue)
|
||||
If imgList.ListExists Then
|
||||
Return imgList.Select(Function(u) New UserMedia(u))
|
||||
Else
|
||||
Return GetGallery(URL, EDP.ReturnValue).ListIfNothing.Select(Function(u) New UserMedia(u))
|
||||
Dim img$ = GetImage(URL, EDP.ReturnValue)
|
||||
If Not img.IsEmptyString Then Return {New UserMedia(img)}
|
||||
End If
|
||||
End If
|
||||
Return Nothing
|
||||
|
||||
@@ -151,10 +151,10 @@ Namespace DownloadObjects
|
||||
Me.Controls.Add(Me.LIST_VIDEOS)
|
||||
Me.Controls.Add(Me.ToolbarBOTTOM)
|
||||
Me.Controls.Add(Me.ToolbarTOP)
|
||||
Me.Icon = Global.SCrawler.My.Resources.Resources.ArrowDownIcon_Blue_24
|
||||
Me.KeyPreview = True
|
||||
Me.MinimumSize = New System.Drawing.Size(540, 400)
|
||||
Me.Name = "VideosDownloaderForm"
|
||||
Me.ShowIcon = False
|
||||
Me.Text = "Download videos"
|
||||
Me.ToolbarTOP.ResumeLayout(False)
|
||||
Me.ToolbarTOP.PerformLayout()
|
||||
|
||||
18
SCrawler/MainFrame.Designer.vb
generated
18
SCrawler/MainFrame.Designer.vb
generated
@@ -121,10 +121,10 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
|
||||
Me.BTT_TRAY_PAUSE_AUTOMATION = New System.Windows.Forms.ToolStripMenuItem()
|
||||
Me.BTT_TRAY_SILENT_MODE = New System.Windows.Forms.ToolStripMenuItem()
|
||||
Me.BTT_TRAY_FEED_SHOW = New System.Windows.Forms.ToolStripMenuItem()
|
||||
Me.BTT_TRAY_CHANNELS = New System.Windows.Forms.ToolStripMenuItem()
|
||||
Me.BTT_TRAY_SHOW_HIDE = New System.Windows.Forms.ToolStripMenuItem()
|
||||
Me.BTT_TRAY_CLOSE = New System.Windows.Forms.ToolStripMenuItem()
|
||||
Me.BTT_TRAY_CLOSE_NO_SCRIPT = New System.Windows.Forms.ToolStripMenuItem()
|
||||
Me.BTT_TRAY_CHANNELS = New System.Windows.Forms.ToolStripMenuItem()
|
||||
SEP_1 = New System.Windows.Forms.ToolStripSeparator()
|
||||
SEP_2 = New System.Windows.Forms.ToolStripSeparator()
|
||||
CONTEXT_SEP_1 = New System.Windows.Forms.ToolStripSeparator()
|
||||
@@ -419,7 +419,7 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
|
||||
'BTT_DOWN_VIDEO
|
||||
'
|
||||
Me.BTT_DOWN_VIDEO.AutoToolTip = True
|
||||
Me.BTT_DOWN_VIDEO.DisplayStyle = System.Windows.Forms.ToolStripItemDisplayStyle.Text
|
||||
Me.BTT_DOWN_VIDEO.Image = Global.SCrawler.My.Resources.Resources.ArrowDownPic_Blue_24
|
||||
Me.BTT_DOWN_VIDEO.ImageTransparentColor = System.Drawing.Color.Magenta
|
||||
Me.BTT_DOWN_VIDEO.Name = "BTT_DOWN_VIDEO"
|
||||
Me.BTT_DOWN_VIDEO.Size = New System.Drawing.Size(231, 22)
|
||||
@@ -856,6 +856,13 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
|
||||
Me.BTT_TRAY_FEED_SHOW.Text = "Feed"
|
||||
Me.BTT_TRAY_FEED_SHOW.ToolTipText = "Show feed of recently downloaded data." & Global.Microsoft.VisualBasic.ChrW(13) & Global.Microsoft.VisualBasic.ChrW(10) & "Ctrl+Click the tray icon to show the feed" &
|
||||
"."
|
||||
'
|
||||
'BTT_TRAY_CHANNELS
|
||||
'
|
||||
Me.BTT_TRAY_CHANNELS.Image = Global.SCrawler.My.Resources.SiteResources.RedditPic_512
|
||||
Me.BTT_TRAY_CHANNELS.Name = "BTT_TRAY_CHANNELS"
|
||||
Me.BTT_TRAY_CHANNELS.Size = New System.Drawing.Size(170, 22)
|
||||
Me.BTT_TRAY_CHANNELS.Text = "Channels"
|
||||
'
|
||||
'BTT_TRAY_SHOW_HIDE
|
||||
'
|
||||
@@ -885,13 +892,6 @@ Partial Public Class MainFrame : Inherits System.Windows.Forms.Form
|
||||
Me.BTT_TRAY_CLOSE_NO_SCRIPT.ToolTipText = "Close the program without executing the script"
|
||||
Me.BTT_TRAY_CLOSE_NO_SCRIPT.Visible = False
|
||||
'
|
||||
'BTT_TRAY_CHANNELS
|
||||
'
|
||||
Me.BTT_TRAY_CHANNELS.Name = "BTT_TRAY_CHANNELS"
|
||||
Me.BTT_TRAY_CHANNELS.Size = New System.Drawing.Size(170, 22)
|
||||
Me.BTT_TRAY_CHANNELS.Text = "Channels"
|
||||
Me.BTT_TRAY_CHANNELS.Image = Global.SCrawler.My.Resources.SiteResources.RedditPic_512
|
||||
'
|
||||
'MainFrame
|
||||
'
|
||||
Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
|
||||
|
||||
@@ -1396,6 +1396,7 @@ ResumeDownloadingOperation:
|
||||
If result < 6 Then
|
||||
Dim collectionResult% = -1
|
||||
Dim tmpResult%
|
||||
Dim tmpUserNames As New List(Of String)
|
||||
Dim IsMultiple As Boolean = users.Count > 1
|
||||
Dim removedUsers As New List(Of String)
|
||||
Dim keepData As Boolean = Not (result Mod 2) = 0
|
||||
@@ -1430,10 +1431,18 @@ ResumeDownloadingOperation:
|
||||
removedUsers.Add(ugn(user))
|
||||
user.Dispose()
|
||||
Else
|
||||
If banUser Then
|
||||
tmpUserNames.Clear()
|
||||
If user.IsCollection Then
|
||||
tmpUserNames.ListAddList(DirectCast(user, UserDataBind).Collections.Select(Function(u) u.Name), l)
|
||||
Else
|
||||
tmpUserNames.Add(user.Name)
|
||||
End If
|
||||
End If
|
||||
tmpResult = user.Delete(IsMultiple, collectionResult)
|
||||
If user.IsCollection And collectionResult = -1 Then collectionResult = tmpResult
|
||||
If tmpResult > 0 Then
|
||||
If banUser Then Settings.BlackList.ListAddValue(New UserBan(user.Name, reason), l) : b = True
|
||||
If banUser And tmpUserNames.Count > 0 Then Settings.BlackList.ListAddList(tmpUserNames.Select(Function(u) New UserBan(u, reason)), l) : b = True
|
||||
RemoveUserFromList(user)
|
||||
removedUsers.Add(ugn(user))
|
||||
Else
|
||||
|
||||
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
|
||||
' by using the '*' as shown below:
|
||||
' <Assembly: AssemblyVersion("1.0.*")>
|
||||
|
||||
<Assembly: AssemblyVersion("2023.1.2.0")>
|
||||
<Assembly: AssemblyFileVersion("2023.1.2.0")>
|
||||
<Assembly: AssemblyVersion("2023.1.24.1")>
|
||||
<Assembly: AssemblyFileVersion("2023.1.24.1")>
|
||||
<Assembly: NeutralResourcesLanguage("en")>
|
||||
|
||||
Reference in New Issue
Block a user