geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-15 00:02:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

2.0.0.0

Browse Source 
Added Instagram downloading, filter by site, channels groups, change folder function, imgur compatibility, special folders, deleting with keeping data, Reddit saved posts downloading
Fixed limited twitter downloading, suspended profiles
Updated download algo
Concat sites editors into a single form
Updated Reddit downloading algo
Fixed saved function in video downloader
Some improvements
This commit is contained in:
Andy
2021-12-27 00:31:03 +03:00
parent 5f2c4476ad
commit 4db7a74e1a
47 changed files with 4682 additions and 2105 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
README.md
View File

@@ -1,8 +1,8 @@
# Social networks crawler
Program for downloading photo and video from Reddit and Twitter
Program for downloading photo and video from Reddit, Twitter and Instagram
Enjoying the tool? Considering adding to my coffee fund :)
Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :)
[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
@@ -12,9 +12,10 @@ Enjoying the tool? Considering adding to my coffee fund :)
- Reddit galleries of images;
- Redgifs hosted videos (https://www.redgifs.com/);
- Reddit hosted videos (downloading Reddit hosted video is going through ffmpeg);
- Twitter images;
- Twitter videos.
- Parse channel and view data.
- Twitter images and videos;
- Instagram images and videos.
- Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels).
- Download [saved Reddit posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts).
- Add users from parsed channel.
- Labeling users.
- Filter exists users by label or group.
@@ -24,41 +25,49 @@ Enjoying the tool? Considering adding to my coffee fund :)
## Reddit
The program parsing all user's posts, gathering pictures' MD5 hash and compare with existing for remove duplicates. Then media will be downloaded.
The program parses all user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
## Twitter
## Twitter and Instagram
The program parsing all user's posts and compare file names with existing for remove duplicates. Then media will be downloaded.
The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
You can read about Instagram restrictions [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-limits)
# Requirements:
- Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher
- Authorization cookies and tokens for Twitter (if you want to download data from Twitter)
- ffmpeg library for download Reddit hosted videos (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip))
- Authorization cookies Instagram (if you want to download data from Instagram)
- ffmpeg library for downloading videos hosted on Reddit (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip))
- Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)
- Just unpack program archive in any folder you want, copy ```ffmpeg.exe``` into and enjoy. :-)
- Just unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy. :)
# Updating
Just download [latest](https://github.com/AAndyProgram/SCrawler/releases/latest) version and unpack it into the program folder. Before starting a new version, I recommend making a backup copy of the program settings folder.
# Settings and usage
The program has an intuitive interface.
Just add user profile and press ```Start downloading``` button.
Just add a user profile and click the ```Start downloading``` button.
Users can be added by patterns:
You can add users by patterns:
- https://www.instagram.com/SomeUserName
- https://twitter.com/SomeUserName
- https://reddit.com/user/SomeUserName
- https://reddit.com/r/SomeSubredditName
- u/SomeUserName
- r/SomeSubredditName
- SomeUserName (in this case you must to choose user site)
- SomeUserName (in this case, you need to select the user's site)
- SomeSubredditName
More about users and subreddits adding [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
Read more about adding users and subreddits [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
**Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
## Using program as just video downloader
Create a shortcut for the program. Open shortcut properties. On the ```Shortcut``` tab in ```Target``` field just add ```v``` at the end through the space.
Create a shortcut for the program. Open shortcut properties. In the ```Shortcut``` tab, in the ```Target``` field, just add the letter ```v``` at the end across the space.
Example: ```D:\Programs\SCrawler\SCrawler.exe v```