geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2023.4.28.0

Browse Source 
Plugins
IPluginContentProvider: added DownloadSingleObject function; added tokens to GetMedia and Download functions; removed GetSpecialData function
Add IDownloadableMedia interface
Removed 'Channel' option from all functions and enums
ISiteSettings: added GetSingleMediaInstance function
ExchangeOptions: removed 'IsChannel'
UserMediaTypes: added Audio and AudioPre enums
IUserMedia, PluginUserMedia: changed ContentType and DownloadState from integers to their enums

SCrawler
Add YouTube standalone downloader
Add gallery-dl & yt-dlp support
Remove 'UserInfo' requirement from 'ProfilesSaved'
Update 'SiteSettingsBase' to use domains and Netscape cookies
UserDataBase: remove channels; remove old 'Merge' const; standardize SavedPosts file naming; move 'ValidateMD5' function from Twitter to UserDataBase to use it in other UserData classes; add 'DownloadSingleObject' environment for single posts; add validating file extension for m3u8 during download; add reindex of video file during download

Rewritten DomainsContainer
Create a universal settings form and PSettingsArttribute
Gfycat, Imgur: turn these classes into IUserData to download a single object

All plugins: update 'GetInstance' function for saved posts; update domains where implemented; remove 'OptionForm' where it exists; update options where they exist; update unix date providers; reconfigure channels where they exist

LPSG: fix attachments; update converters and regex
Add sites: ThisVid, Mastodon, Pinterest, YouTube, YouTube music
Reddit: standardize container parsing for all data types; new channel environment; fix 'ReparseMissing' function; redirect data downloading to the base download function, saved crossposts support
Twitter: fixed gif path bug; fixed downloading saved posts
PornHub: hide unnecessary errors; photo galleries bug
RedGifs: add 'UserAgent' option

Added icons to download progress

Rename some objects
Completely redesigned standalone downloader form and rewritten its environment
WebClient2: update to use tokens

Labels: update label form (save labels to file only when OK button is clicked); change removing labels.txt from recycle bin to permanent; disable storing label 'NoParsedUser'

UserCreatorForm: remove the 'Channel' checkbox and related functions; ability to extract the user's URL from the buffer and apply parameters if found
Remove temporary 'EncryptCookies' module

MainFrame: added simplified way to create new users (Ctrl+Insert to create a new user with default parameters from clipboard URL); removed SCrawler command line argument "-v" (remove the ability to run SCrawler as video downloader)
PropertyValueHost: update for option forms compatibility
SettingsHost: removed 'GetSpecialData' fork; added 'GetSingleMediaInstance' fork
UserDataHost: update functions with tokens; update events; add 'DownloadSingleObject' function
Settings: add the ability to get environment from 4 destinations; add the ability to set the program environment manually; add CMDEncoding; add cache; remove the old function 'RemoveUnusedPlugins'; add 'STDownloader' properties; add YT compatibility; add new notification options; add deleting user settings file when 'SettingsCLS.Dispose()' if where are no users in SCrawler
UserFinder: remove old 'Merge' const; remove channel option
UserInfo: remove channel option
This commit is contained in:
Andy
2023-04-28 10:13:46 +03:00
parent db9e2cfb88
commit b2a9b22478
270 changed files with 18120 additions and 3332 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
README.md
View File

@@ -6,32 +6,39 @@
[![FAQ](https://img.shields.io/badge/FAQ-green)](FAQ.md)
[![GUIDE](https://img.shields.io/badge/GUIDE-green)](https://github.com/AAndyProgram/SCrawler/wiki)
[![How to support](https://img.shields.io/badge/HowToSupport-green)](HowToSupport.md)
:eu:
:greece:
A program to download photo and video from [any site](#supported-sites) (e.g. Reddit, Twitter, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, LPSG).
A program to download photo and video from [any site](#supported-sites) (e.g. YouTube, YouTube Music, Reddit, Twitter, Mastodon, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, ThisVid, LPSG, Pinterest).
**If you like SCrawler, please like the program on [this site](https://alternativeto.net/software/scrawler/about/) and/or [this](https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml)**
Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :blush:
[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
<!---Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :blush:
[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)--->
**Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
![Main window](ProgramScreenshots/MainWindow.png)
![Channels window](ProgramScreenshots/Channels.png)
[**YouTube standalone application:**](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader)
![YouTube application](ProgramScreenshots/AppYouTube.png)
# What can program do:
- Download pictures and videos from users' profiles and subreddits:
- YouTube videos, shorts, users, artists, playlists, music, tracks;
- Reddit images, galleries of images, videos, saved posts;
- Redgifs videos (https://www.redgifs.com/);
- Twitter images and videos, saved (bookmarked) posts;
- Mastodon images and videos, saved (bookmarked) posts;
- Instagram images and videos, tagged posts, stories, saved posts;
- TikTok videos ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits));
- TikTok videos (*currently broken*; [limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits));
- Pinterest boards, users, saved posts;
- Imgur images, galleries and videos;
- Gfycat videos;
- PornHub images, videos, save (liked) posts;
- XHamster images, videos, saved posts;
- XVIDEOS videos;
- XVIDEOS videos, saved posts;
- ThiVid images, videos, saved posts;
- [Other](#supported-sites) supported sites
- Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
- Download [saved Reddit, Twitter and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
@@ -52,18 +59,22 @@ Do you like this program? Consider adding to my coffee fund by making a donation
- ...and many others...
# Supported sites
- **YouTube**
- **YouTube Music**
- **Reddit**
- **Twitter**
- **Mastodon**
- **Instagram**
- **TikTok** ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
- TikTok (*currently broken*; [limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
- RedGifs
- Pinterest
- Imgur
- Gfycat
- LPSG
- **PornHub**
- **XHamster**
- **XVIDEOS**
- **ThisVid**
- [Other sites](Plugins.md)
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
@@ -82,7 +93,8 @@ The program parses user posts and compares file names with existing ones to remo
## How to request a new site
Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
<!---Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about--->
**I'm currently not accepting requests to develop new sites.**
# Requirements
@@ -108,12 +120,16 @@ Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
- [Reddit](https://github.com/AAndyProgram/SCrawler/wiki/Settings#reddit)
- [Twitter](https://github.com/AAndyProgram/SCrawler/wiki/Settings#twitter)
- [Mastodon](https://github.com/AAndyProgram/SCrawler/wiki/Settings#Mastodon)
- [Instagram](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram)
- [TikTok](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok)
- [RedGifs](https://github.com/AAndyProgram/SCrawler/wiki/Settings#redgifs)
- [YouTube](https://github.com/AAndyProgram/SCrawler/wiki/Settings#YouTube)
- [Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#Pinterest)
- [PornHub](https://github.com/AAndyProgram/SCrawler/wiki/Settings#pornhub)
- [XHamster](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xhamster)
- [XVIDEOS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xvideos)
- [ThisVid](https://github.com/AAndyProgram/SCrawler/wiki/Settings#ThisVid)
- [LPSG](https://github.com/AAndyProgram/SCrawler/wiki/Settings#lpsg)
**Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
@@ -142,20 +158,11 @@ The program has an intuitive interface.
Just add a user profile and **click the ```Download``` button**.
Read more about adding users and subreddits [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
Read more about adding users and subreddits [here](https://github.com/AAndyProgram/SCrawler/wiki#Add%20user)
![Add user](ProgramScreenshots/CreateUserClear.png)
# Using program as just video downloader
Create a shortcut for the program. Open shortcut properties. In the ```Shortcut``` tab, in the ```Target``` field, just add the letter ```v``` at the end across the space.
Example: ```D:\Programs\SCrawler\SCrawler.exe v```
![Separate video downloader](ProgramScreenshots/SeparateVideoDownloader.png)
# Contact me
[![matrix](https://img.shields.io/badge/Matrix-%40andyprogram%3Amatrix.org-informational)](https://matrix.to/#/@andyprogram:matrix.org)
[![discord](https://img.shields.io/badge/discord-AndyProgram%233804-yellowgreen)](https://discordapp.com/users/1012768226679206009) AndyProgram#3804
Matrix (Element): https://matrix.to/#/@andyprogram:matrix.org
Discord: AndyProgram#3804