mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-14 15:52:18 +00:00
67c4b25d22
Add full channel support, make ready for download prop available for collections and multiusers Fixed imgur pics compatibility Updated abstract classes and functions for channels compatibility
65 lines
2.5 KiB
Markdown
65 lines
2.5 KiB
Markdown
# Social networks crawler
|
|
|
|
Program for downloading photo and video from Reddit and Twitter
|
|
|
|
Enjoying the tool? Considering adding to my coffee fund :)
|
|
|
|
[](https://ko-fi.com/andyprogram)
|
|
|
|
# What can program do:
|
|
- Download pictures and videos from users' profiles and subreddits:
|
|
- Reddit images;
|
|
- Reddit galleries of images;
|
|
- Redgifs hosted videos (https://www.redgifs.com/);
|
|
- Reddit hosted videos (downloading Reddit hosted video is going through ffmpeg);
|
|
- Twitter images;
|
|
- Twitter videos.
|
|
- Parse channel and view data.
|
|
- Add users from parsed channel.
|
|
- Labeling users.
|
|
- Filter exists users by label or group.
|
|
- Selection of media types you want to download (images only, videos only, both)
|
|
|
|
# How does it works:
|
|
|
|
## Reddit
|
|
|
|
The program parsing all user's posts, gathering pictures' MD5 hash and compare with existing for remove duplicates. Then media will be downloaded.
|
|
|
|
## Twitter
|
|
|
|
The program parsing all user's posts and compare file names with existing for remove duplicates. Then media will be downloaded.
|
|
|
|
# Requirements:
|
|
|
|
- Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher
|
|
- Authorization cookies and tokens for Twitter (if you want to download data from Twitter)
|
|
- ffmpeg library for download Reddit hosted videos (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip))
|
|
- Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)
|
|
- Just unpack program archive in any folder you want, copy ```ffmpeg.exe``` into and enjoy. :-)
|
|
|
|
# Settings and usage
|
|
|
|
The program has an intuitive interface.
|
|
|
|
Just add user profile and press ```Start downloading``` button.
|
|
|
|
Users can be added by patterns:
|
|
- https://twitter.com/SomeUserName
|
|
- https://reddit.com/user/SomeUserName
|
|
- https://reddit.com/r/SomeSubredditName
|
|
- u/SomeUserName
|
|
- r/SomeSubredditName
|
|
- SomeUserName (in this case you must to choose user site)
|
|
- SomeSubredditName
|
|
|
|
More about users and subreddits adding [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
|
|
|
|
**Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
|
|
|
|
## Using program as just video downloader
|
|
|
|
Create a shortcut for the program. Open shortcut properties. On the ```Shortcut``` tab in ```Target``` field just add ```v``` at the end through the space.
|
|
|
|
Example: ```D:\Programs\SCrawler\SCrawler.exe v```
|