mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-15 16:22:17 +00:00
Compare commits
3 Commits
2023.1.2.0
...
2023.1.24.
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
602771d982 | ||
|
|
3e472b4f2b | ||
|
|
30c3fe3b68 |
@@ -8,14 +8,16 @@ I welcome requests! Follow these steps to contribute:
|
||||
1. If you have a code change suggestion, you can post a replacement code block. I also accept pull requests.
|
||||
|
||||
# How to build from source
|
||||
1. Delete the "PersonalUtilities" project from the solution.
|
||||
1. Delete the "PersonalUtilities.Notifications" project from the solution.
|
||||
1. Delete the ```PersonalUtilities``` project from the solution.
|
||||
1. Delete the ```PersonalUtilities.Notifications``` project from the solution.
|
||||
1. Delete the ```cURL``` folder from the solution.
|
||||
1. Delete the ```ffmpeg.exe``` from the solution.
|
||||
1. The following libraries must be added to project references with the '**Copy to output folder**' option:
|
||||
- ```PersonalUtilities.dll```
|
||||
- ```PersonalUtilities.Notifications.dll```
|
||||
- ```Microsoft.Toolkit.Uwp.Notifications.dll```
|
||||
- ```System.ValueTuple.dll```
|
||||
1. Import PersonalUtilities.Functions for the whole project.
|
||||
1. Import ```PersonalUtilities.Functions``` for the whole project.
|
||||
|
||||
**Always use the correct libraries. You must download libraries from the same release date as the code commit date.**
|
||||
|
||||
|
||||
@@ -1,3 +1,11 @@
|
||||
# 2023.1.24.0
|
||||
|
||||
*2023-01-24*
|
||||
|
||||
- Fixed
|
||||
- (Issue #100) Imgur albums not downloading
|
||||
- When deleting a collection with the 'ban' option, users in the collection are not banned
|
||||
|
||||
# 2023.1.2.0
|
||||
|
||||
*2023-01-02*
|
||||
|
||||
@@ -6,6 +6,7 @@ You can support the program by:
|
||||
- :repeat: make a post about my program on your profile (Reddit, Twitter, Instagram and any other social networks)
|
||||
- :speech_balloon: tell your friends about the program
|
||||
- :heart: like the program on this site: https://alternativeto.net/software/scrawler/about/
|
||||
- :heart: like the program on this site: https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml
|
||||
- suggest my program as an alternative ([on this site](https://alternativeto.net/software/scrawler/about/)) to any program you have used before
|
||||
|
||||
I would be very grateful for any support! :blush:
|
||||
|
||||
26
README.md
26
README.md
@@ -1,6 +1,4 @@
|
||||
# :rainbow_flag: Social networks crawler :rainbow_flag: :christmas_tree:
|
||||
|
||||
# :christmas_tree: Happy new year :christmas_tree:
|
||||
# :rainbow_flag: Social networks crawler :rainbow_flag:
|
||||
|
||||
[](https://github.com/AAndyProgram/SCrawler/releases/latest)
|
||||
[](https://github.com/AAndyProgram/SCrawler/blob/main/LICENSE)
|
||||
@@ -39,18 +37,18 @@ Do you like this program? Consider adding to my coffee fund by making a donation
|
||||
- Download [saved Reddit, Twitter and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
|
||||
- Add users from parsed channel
|
||||
- **Advanced user management**
|
||||
- **Automation** (downloading data automatically every ```X``` minutes)
|
||||
- **Feed** (feed of downloaded media files)
|
||||
- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every ```X``` minutes)
|
||||
- **Feed** ([feed](https://github.com/AAndyProgram/SCrawler/wiki#feed) of downloaded media files)
|
||||
- Labeling users
|
||||
- Create download groups
|
||||
- Create [download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
|
||||
- Adding users to favorites and temporary
|
||||
- Filter exists users by label or group
|
||||
- [Filter exists users](https://github.com/AAndyProgram/SCrawler/wiki#view) by label or group
|
||||
- Selection of media types you want to download (images only, videos only, both)
|
||||
- Download a special video, image or gallery
|
||||
- Making collections (grouping users into collections)
|
||||
- [Download a special video](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video), image or gallery
|
||||
- Making [collections](https://github.com/AAndyProgram/SCrawler/wiki#collections) (grouping users into collections)
|
||||
- Specifying a user folder (for downloading data to another location)
|
||||
- Changing user icons
|
||||
- Changing view modes
|
||||
- Changing [view modes](https://github.com/AAndyProgram/SCrawler/wiki#view)
|
||||
- ...and many others...
|
||||
|
||||
# Supported sites
|
||||
@@ -76,11 +74,11 @@ First, the program downloads the full profile. After the program downloads only
|
||||
|
||||
## Reddit
|
||||
|
||||
The program parses all user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
|
||||
The program parses user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
|
||||
|
||||
## Other sites
|
||||
|
||||
The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
|
||||
The program parses user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
|
||||
|
||||
## How to request a new site
|
||||
|
||||
@@ -122,7 +120,7 @@ Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
|
||||
|
||||
# Installation
|
||||
|
||||
**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy.** :blush:
|
||||
**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder and enjoy.** :blush:
|
||||
|
||||
**Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
|
||||
|
||||
@@ -140,7 +138,7 @@ Read about how to make plugin [here](https://github.com/AAndyProgram/SCrawler/wi
|
||||
|
||||
# How to support
|
||||
|
||||
Read more about how to support the program [here](HowToSupport.md).
|
||||
Read about how to support the program [here](HowToSupport.md).
|
||||
|
||||
# Settings and usage
|
||||
|
||||
|
||||
@@ -70,11 +70,12 @@ Namespace API.Imgur
|
||||
Friend Shared Function GetVideoInfo(ByVal URL As String, Optional ByVal e As ErrorsDescriber = Nothing) As IEnumerable(Of UserMedia)
|
||||
Try
|
||||
If Not URL.IsEmptyString AndAlso URL.ToLower.Contains("imgur") AndAlso Not Settings.ImgurClientID.IsEmptyString Then
|
||||
Dim img$ = GetImage(URL, EDP.ReturnValue)
|
||||
If Not img.IsEmptyString Then
|
||||
Return {New UserMedia(img)}
|
||||
Dim imgList As List(Of String) = GetGallery(URL, EDP.ReturnValue)
|
||||
If imgList.ListExists Then
|
||||
Return imgList.Select(Function(u) New UserMedia(u))
|
||||
Else
|
||||
Return GetGallery(URL, EDP.ReturnValue).ListIfNothing.Select(Function(u) New UserMedia(u))
|
||||
Dim img$ = GetImage(URL, EDP.ReturnValue)
|
||||
If Not img.IsEmptyString Then Return {New UserMedia(img)}
|
||||
End If
|
||||
End If
|
||||
Return Nothing
|
||||
|
||||
@@ -1396,6 +1396,7 @@ ResumeDownloadingOperation:
|
||||
If result < 6 Then
|
||||
Dim collectionResult% = -1
|
||||
Dim tmpResult%
|
||||
Dim tmpUserNames As New List(Of String)
|
||||
Dim IsMultiple As Boolean = users.Count > 1
|
||||
Dim removedUsers As New List(Of String)
|
||||
Dim keepData As Boolean = Not (result Mod 2) = 0
|
||||
@@ -1430,10 +1431,18 @@ ResumeDownloadingOperation:
|
||||
removedUsers.Add(ugn(user))
|
||||
user.Dispose()
|
||||
Else
|
||||
If banUser Then
|
||||
tmpUserNames.Clear()
|
||||
If user.IsCollection Then
|
||||
tmpUserNames.ListAddList(DirectCast(user, UserDataBind).Collections.Select(Function(u) u.Name), l)
|
||||
Else
|
||||
tmpUserNames.Add(user.Name)
|
||||
End If
|
||||
End If
|
||||
tmpResult = user.Delete(IsMultiple, collectionResult)
|
||||
If user.IsCollection And collectionResult = -1 Then collectionResult = tmpResult
|
||||
If tmpResult > 0 Then
|
||||
If banUser Then Settings.BlackList.ListAddValue(New UserBan(user.Name, reason), l) : b = True
|
||||
If banUser And tmpUserNames.Count > 0 Then Settings.BlackList.ListAddList(tmpUserNames.Select(Function(u) New UserBan(u, reason)), l) : b = True
|
||||
RemoveUserFromList(user)
|
||||
removedUsers.Add(ugn(user))
|
||||
Else
|
||||
|
||||
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
|
||||
' by using the '*' as shown below:
|
||||
' <Assembly: AssemblyVersion("1.0.*")>
|
||||
|
||||
<Assembly: AssemblyVersion("2023.1.2.0")>
|
||||
<Assembly: AssemblyFileVersion("2023.1.2.0")>
|
||||
<Assembly: AssemblyVersion("2023.1.24.0")>
|
||||
<Assembly: AssemblyFileVersion("2023.1.24.0")>
|
||||
<Assembly: NeutralResourcesLanguage("en")>
|
||||
|
||||
Reference in New Issue
Block a user