mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-14 15:52:18 +00:00
b2a9b22478
Plugins IPluginContentProvider: added DownloadSingleObject function; added tokens to GetMedia and Download functions; removed GetSpecialData function Add IDownloadableMedia interface Removed 'Channel' option from all functions and enums ISiteSettings: added GetSingleMediaInstance function ExchangeOptions: removed 'IsChannel' UserMediaTypes: added Audio and AudioPre enums IUserMedia, PluginUserMedia: changed ContentType and DownloadState from integers to their enums SCrawler Add YouTube standalone downloader Add gallery-dl & yt-dlp support Remove 'UserInfo' requirement from 'ProfilesSaved' Update 'SiteSettingsBase' to use domains and Netscape cookies UserDataBase: remove channels; remove old 'Merge' const; standardize SavedPosts file naming; move 'ValidateMD5' function from Twitter to UserDataBase to use it in other UserData classes; add 'DownloadSingleObject' environment for single posts; add validating file extension for m3u8 during download; add reindex of video file during download Rewritten DomainsContainer Create a universal settings form and PSettingsArttribute Gfycat, Imgur: turn these classes into IUserData to download a single object All plugins: update 'GetInstance' function for saved posts; update domains where implemented; remove 'OptionForm' where it exists; update options where they exist; update unix date providers; reconfigure channels where they exist LPSG: fix attachments; update converters and regex Add sites: ThisVid, Mastodon, Pinterest, YouTube, YouTube music Reddit: standardize container parsing for all data types; new channel environment; fix 'ReparseMissing' function; redirect data downloading to the base download function, saved crossposts support Twitter: fixed gif path bug; fixed downloading saved posts PornHub: hide unnecessary errors; photo galleries bug RedGifs: add 'UserAgent' option Added icons to download progress Rename some objects Completely redesigned standalone downloader form and rewritten its environment WebClient2: update to use tokens Labels: update label form (save labels to file only when OK button is clicked); change removing labels.txt from recycle bin to permanent; disable storing label 'NoParsedUser' UserCreatorForm: remove the 'Channel' checkbox and related functions; ability to extract the user's URL from the buffer and apply parameters if found Remove temporary 'EncryptCookies' module MainFrame: added simplified way to create new users (Ctrl+Insert to create a new user with default parameters from clipboard URL); removed SCrawler command line argument "-v" (remove the ability to run SCrawler as video downloader) PropertyValueHost: update for option forms compatibility SettingsHost: removed 'GetSpecialData' fork; added 'GetSingleMediaInstance' fork UserDataHost: update functions with tokens; update events; add 'DownloadSingleObject' function Settings: add the ability to get environment from 4 destinations; add the ability to set the program environment manually; add CMDEncoding; add cache; remove the old function 'RemoveUnusedPlugins'; add 'STDownloader' properties; add YT compatibility; add new notification options; add deleting user settings file when 'SettingsCLS.Dispose()' if where are no users in SCrawler UserFinder: remove old 'Merge' const; remove channel option UserInfo: remove channel option
🏳️🌈 Social networks crawler 🏳️🌈
A program to download photo and video from any site (e.g. YouTube, YouTube Music, Reddit, Twitter, Mastodon, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, ThisVid, LPSG, Pinterest).
If you like SCrawler, please like the program on this site and/or this
Bitcoin: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
YouTube standalone application:
What can program do:
- Download pictures and videos from users' profiles and subreddits:
- YouTube videos, shorts, users, artists, playlists, music, tracks;
- Reddit images, galleries of images, videos, saved posts;
- Redgifs videos (https://www.redgifs.com/);
- Twitter images and videos, saved (bookmarked) posts;
- Mastodon images and videos, saved (bookmarked) posts;
- Instagram images and videos, tagged posts, stories, saved posts;
- TikTok videos (currently broken; limited);
- Pinterest boards, users, saved posts;
- Imgur images, galleries and videos;
- Gfycat videos;
- PornHub images, videos, save (liked) posts;
- XHamster images, videos, saved posts;
- XVIDEOS videos, saved posts;
- ThiVid images, videos, saved posts;
- Other supported sites
- Parse channel and view data
- Download saved Reddit, Twitter and Instagram posts
- Add users from parsed channel
- Advanced user management
- Automation (downloading data automatically every
Xminutes) - Feed (feed of downloaded media files)
- Labeling users
- Create download groups
- Adding users to favorites and temporary
- Filter exists users by label or group
- Selection of media types you want to download (images only, videos only, both)
- Download a special video, image or gallery
- Making collections (grouping users into collections)
- Specifying a user folder (for downloading data to another location)
- Changing user icons
- Changing view modes
- ...and many others...
Supported sites
- YouTube
- YouTube Music
- Mastodon
- TikTok (currently broken; limited)
- RedGifs
- Imgur
- Gfycat
- LPSG
- PornHub
- XHamster
- XVIDEOS
- ThisVid
- Other sites
How it works
First, the program downloads the full profile. After the program downloads only new posts. The program remembers downloaded posts.
The program parses user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
Other sites
The program parses user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
How to request a new site
I'm currently not accepting requests to develop new sites.
Requirements
- Windows 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this tool.
- SITES REQUIREMENTS
Guide
Full guide you can find here
Installation
Just download the latest release, unzip the program archive to any folder and enjoy. 😊
Don't put program in the Program Files system folder (this is portable program and program settings are stored in the program folder)
Updating
Just download latest version and unpack it into the program folder. Before starting a new version, I recommend making a backup copy of the program settings folder.
How to build from source
How to make a plugin
How to support
Settings and usage
The program has an intuitive interface.
Just add a user profile and click the Download button.
Read more about adding users and subreddits here
Contact me
Matrix (Element): https://matrix.to/#/@andyprogram:matrix.org Discord: AndyProgram#3804