mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-14 15:52:18 +00:00
df06a86651
Plugins.Attributes: add 'DependentFields' attribute Plugins.IPluginContentProvider: add 'Options' and 'IsSubscription' properties Plugins.ISiteSettings: add 'SubscriptionsAllowed' property Plugins.ExchangeOptions: add 'Options' field Plugins.Attributes.PropertyUpdater: replace 'Dependencies' with 'Arguments' YT: add 'OutputPathAskForName' and 'OutputPathAutoAddPaths' properties; add the ability to store download locations; add 'DownloadLocation' and 'DownloadLocationsCollection' objects YT.IDownloaderSettings: add 'OutputPathAskForName' and 'OutputPathAutoAddPaths' properties YT.Downloader: fixed bug with re-saving elements when loading a video list; fixed bug when files were not deleted when clicking on the delete button; fixed a bug that caused the video to redownload; download job removes elements at wrong indexes; added skipping of downloaded elements in the job; fixed a bug, pending option did not change after download complete YT.YouTubeMediaContainerBase: add '_MediaStateOnLoad' field and 'NeedToSave' function; update the 'Save' function to prevent saving a file when a download is complete and the file has already been saved; update code for new yt-dlp version Fixed cache deletion errors Add user queue Add global locations API.Base.SiteSettingsBase: implement 'SubscriptionsAllowed' property; remove request headers with null values on save; add '_AllowUserAgentUpdate' parameter API.Base.Structures: add 'SiteModes' enum API.Base.UserDataBase: add 'Erase' button; implement 'Options' and 'IsSubscription' properties; add 'SpecialLabels' property; update 'LVIKey'; update 'FitToAddParams' function; add 'EraseData' function; user colors; Not UserExists notification, UserQueue support API.Base: add 'DeclaredNames' API.Instagram: remove default values for headers; disable updating UserAgent from global; check for a new username for non-existent users API.Mastodon: bypass new inherited twitter options; update names and headers API.OnlyFans: make 'HH_BROWSER' property nullable; remove 'HH_BROWSER' from required; fix username bug (dots); handling of 504 and 429 errors; add 'DownloadHighlights' and 'DownloadChatMedia' options; add 'UserExchangeOptions'; fixed incorrect error handler API.PathPlugin: fixed incorrect detection of path existence API.Pinterest: add 'SpecialLabels' API.PornHub: add new video regex; remove old regex; added 'DownloadUploaded', 'DownloadTagged', 'DownloadPrivate' and 'DownloadFavorite' properties to 'SiteSettings', 'UserData' and 'UserExchangeOptions'; update regex to define user; added downloading search queries; update 'GetUserUrl' function; hide unnecessary 'RegexFieldsTextBecameNullException' errors; add subscriptions API.Reddit: add 'SpecialLabels'; add bearer token and its refresh interval; add OAuth; add additional options API.RedGifs: add 'DependentFields' for 'Token' API.ThisVid: add 'DownloadFavourite' option; add downloading search queries, tags, categories; add 'SpecialLabels'; add subscriptions; updating cookies issue API.TikTok: rewrite algorithms API.Twitter: add 'UseAppropriateModel', 'UseNewEndPointSearch', 'UseNewEndPointProfiles', 'AbortOnLimit', 'DownloadAlreadyParsed', 'MediaModelAllowNonUserTweets' properties; remove old commented code; remove 'TwitterPic_400' and replace with 'TwitterIcon_32.ToBitmap'; add 'DownloadModelForceApply' user option; update environment to GDL 1.25.8; fixed gifs downloading; fix typo in 'ReparseMissing'; update names API.UserDataBind: prevent adding site-specific labels when adding to a collection API.Xhamster: add downloading search queries, tags, categories; add 'SpecialLabels'; add additional nodes for channels; add subscriptions API.XVIDEOS: add downloading search queries, tags, categories; add 'SpecialLabels'; add subscriptions; changed users creation method; add subscriptions API.YouTube: add subscriptions AutoDownloader: add new group subscription options; update predicates; fixed excluded labels and sites in default mode; update notifications; add an additional skip options, add 'Force start' option DownloadedInfoForm: add subscriptions; fixed size/location bug; hide unnecessary error (refill) Feed: add subscriptions; update filters; add 'Ctrl+G' shortcut FeedMedia: add subscriptions; fixed 'webm' bug; add title for subscription media; add site icon to post; user colors; always using 'FriendlyName' instead of 'UserName' if it exists DownloadGroup, GroupDefaults, GroupParameters: add subscription and 'UsersCount' options MissingPostsForm: add 'BTT_DELETE_ALL' VideoDownloaderForm, DownloaderUrlForm, DownloaderUrlsArrForm: add download locations support VideoDownloaderForm: add subscriptions support GlobalSettingsForm: add new properties UserCreatorForm: add subscriptions; add 'Options' support (of 'ExchangeOptions'); user colors ListImagesLoader: add subscription colors; user colors MainFrame: add subscriptions; add filters by subscription and user; update predicates NuGet: update 'LibVLCSharp', 'LibVLCSharp.WinForms', 'VideoLAN.LibVLC.Windows' DownloadableMediaHost: update 'Save' function PropertyValueHost: fix 'CaptionWidth' bug; add 'Dependents' SettingsHost: add 'Dependents' UserDataHost: add 'Options' and 'IsSubscription' properties SettingsCLS: implement new 'IDownloaderSettings' properties; add 'CacheSnapshots'; add 'DownloadLocations'; add new properties UserInfo, UserFinder: add subscriptions UserSearchForm: fixed search by name bug
206 lines
10 KiB
Markdown
206 lines
10 KiB
Markdown
# :rainbow_flag: Happy LGBT Pride Month :tada:
|
|
|
|
# :rainbow_flag: Social networks crawler :rainbow_flag:
|
|
|
|
[](https://github.com/AAndyProgram/SCrawler/releases/latest)
|
|
[](https://github.com/AAndyProgram/SCrawler/blob/main/LICENSE)
|
|
[](https://github.com/AAndyProgram/SCrawler/releases)
|
|
[](FAQ.md)
|
|
[](https://github.com/AAndyProgram/SCrawler/wiki)
|
|
[](HowToSupport.md)
|
|
:eu:
|
|
:greece:
|
|
|
|
A program to download photo and video from [any site](#supported-sites) (e.g. YouTube, YouTube Music, OnlyFans, Reddit, Twitter, Mastodon, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, ThisVid, LPSG, Pinterest).
|
|
|
|
**If you like SCrawler, please like the program on [this site](https://alternativeto.net/software/scrawler/about/) and/or [this](https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml)**
|
|
<!---Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :blush:
|
|
[](https://ko-fi.com/andyprogram)--->
|
|
**Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
|
|
|
|
[](https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml#status)
|
|
|
|
|
|
|
|
|
|
[**YouTube standalone application:**](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader)
|
|
|
|
|
|
|
|
# What can program do:
|
|
- Download pictures and videos from users' profiles and subreddits:
|
|
- YouTube videos, shorts, users, artists, playlists, music, tracks;
|
|
- Reddit images, galleries of images, videos, saved posts;
|
|
- Redgifs videos (https://www.redgifs.com/);
|
|
- Twitter images and videos, saved (bookmarked) posts;
|
|
- OnlyFans images and videos, saved (bookmarked) posts;
|
|
- Mastodon images and videos, saved (bookmarked) posts;
|
|
- Instagram images and videos, tagged posts, stories, saved posts;
|
|
- TikTok videos (*currently broken*; [limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits));
|
|
- Pinterest boards, users, saved posts;
|
|
- Imgur images, galleries and videos;
|
|
- Gfycat videos;
|
|
- PornHub images, videos, save (liked) posts, search queries, search categories;
|
|
- XHamster images, videos, saved posts, search queries, search categories, search tags;
|
|
- XVIDEOS videos, saved posts, search queries, search categories;
|
|
- ThisVid images, videos, saved posts, search queries, search categories, search tags;
|
|
- [Other](#supported-sites) supported sites
|
|
- Parse [Reddit channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
|
|
- Download [saved posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
|
|
- Add users from parsed channel
|
|
- **Advanced user management**
|
|
- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every ```X``` minutes)
|
|
- **Feed** ([feed](https://github.com/AAndyProgram/SCrawler/wiki#feed) of downloaded media files and subscriptions posts)
|
|
- Labeling users
|
|
- Create [download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
|
|
- Adding users to favorites and temporary
|
|
- Adding users and search queries in the **Subscription** mode (download post preview, but do not download the media file)
|
|
- [Filter exists users](https://github.com/AAndyProgram/SCrawler/wiki#view) by label or group
|
|
- Selection of media types you want to download (images only, videos only, both)
|
|
- [Download a special video](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video), image or gallery
|
|
- Making [collections](https://github.com/AAndyProgram/SCrawler/wiki#collections) (grouping users into collections)
|
|
- Specifying a user folder (for downloading data to another location)
|
|
- Changing user icons
|
|
- Changing [view modes](https://github.com/AAndyProgram/SCrawler/wiki#view)
|
|
- ...and many others...
|
|
|
|
# Supported sites
|
|
- **YouTube**
|
|
- **YouTube Music**
|
|
- **Reddit**
|
|
- **Twitter**
|
|
- **OnlyFans**
|
|
- **Mastodon**
|
|
- **Instagram**
|
|
- TikTok
|
|
- RedGifs
|
|
- Pinterest
|
|
- Imgur
|
|
- Gfycat
|
|
- LPSG
|
|
- **PornHub**
|
|
- **XHamster**
|
|
- **XVIDEOS**
|
|
- **ThisVid**
|
|
- [Other sites](Plugins.md)
|
|
|
|
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
|
|
|
|
# How it works
|
|
|
|
First, the program downloads the full profile. After the program downloads only new posts. The program remembers downloaded posts.
|
|
|
|
## Reddit
|
|
|
|
The program parses user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
|
|
|
|
## Other sites
|
|
|
|
The program parses user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
|
|
|
|
## How to request a new site
|
|
|
|
<!---Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about--->
|
|
**I'm currently not accepting requests to develop new sites.**
|
|
|
|
# Requirements
|
|
|
|
- Windows 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
|
|
- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
|
|
|
|
# Guide
|
|
|
|
- [Main window](https://github.com/AAndyProgram/SCrawler/wiki)
|
|
- [Users](https://github.com/AAndyProgram/SCrawler/wiki/Users)
|
|
- [Add/Edit/Delete users](https://github.com/AAndyProgram/SCrawler/wiki/Users)
|
|
- [Collections](https://github.com/AAndyProgram/SCrawler/wiki#collections)
|
|
- [User operations](https://github.com/AAndyProgram/SCrawler/wiki#context-menu)
|
|
- [User labels](https://github.com/AAndyProgram/SCrawler/wiki/Users#labels)
|
|
- **[DOWNLOAD](https://github.com/AAndyProgram/SCrawler/wiki#download)**
|
|
- [Automation](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)
|
|
- [Download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
|
|
- [Downloading information](https://github.com/AAndyProgram/SCrawler/wiki#info)
|
|
- [Reddit channels](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
|
|
- [Saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)
|
|
- [View modes, filters](https://github.com/AAndyProgram/SCrawler/wiki#view)
|
|
- **[SETTINGS](https://github.com/AAndyProgram/SCrawler/wiki/Settings)**
|
|
- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
|
|
- [Reddit](https://github.com/AAndyProgram/SCrawler/wiki/Settings#reddit)
|
|
- [Twitter](https://github.com/AAndyProgram/SCrawler/wiki/Settings#twitter)
|
|
- [OnlyFans](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans)
|
|
- [Mastodon](https://github.com/AAndyProgram/SCrawler/wiki/Settings#Mastodon)
|
|
- [Instagram](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram)
|
|
- [TikTok](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok)
|
|
- [RedGifs](https://github.com/AAndyProgram/SCrawler/wiki/Settings#redgifs)
|
|
- [YouTube](https://github.com/AAndyProgram/SCrawler/wiki/Settings#YouTube)
|
|
- [Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#Pinterest)
|
|
- [PornHub](https://github.com/AAndyProgram/SCrawler/wiki/Settings#pornhub)
|
|
- [XHamster](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xhamster)
|
|
- [XVIDEOS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xvideos)
|
|
- [ThisVid](https://github.com/AAndyProgram/SCrawler/wiki/Settings#ThisVid)
|
|
- [LPSG](https://github.com/AAndyProgram/SCrawler/wiki/Settings#lpsg)
|
|
|
|
**Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
|
|
|
|
# Installation
|
|
|
|
**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder and enjoy.** :blush:
|
|
|
|
**Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
|
|
|
|
# Updating
|
|
|
|
Just download [latest](https://github.com/AAndyProgram/SCrawler/releases/latest) version and unpack it into the program folder. **Before starting a new version, I recommend making a backup copy of the program settings folder.**
|
|
|
|
# [How to build from source](CONTRIBUTING.md#how-to-build-from-source)
|
|
|
|
# [How to make a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins)
|
|
|
|
# [How to support](HowToSupport.md)
|
|
|
|
# Settings and usage
|
|
|
|
The program has an intuitive interface.
|
|
|
|
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
|
|
|
|
Just add a user profile and **click the ```Download``` button**.
|
|
|
|
```mermaid
|
|
stateDiagram
|
|
Start: Add site credentials
|
|
What: What would I like to do
|
|
DownUser: Download user
|
|
DownVideo: Download video
|
|
AUser: Add user (1)
|
|
OVIF: Open standalone downloader (2)
|
|
AVideo: Add video url
|
|
F5: Press 'F5' or click the download button
|
|
[*]-->Start
|
|
Start-->What
|
|
What-->DownUser
|
|
What-->DownVideo
|
|
DownUser-->AUser
|
|
DownVideo-->OVIF
|
|
OVIF-->AVideo
|
|
AVideo-->F5
|
|
AUser-->F5
|
|
F5-->[*]
|
|
```
|
|
1. Press `Insert` or click the `Download` button ([read more here](https://github.com/AAndyProgram/SCrawler/wiki#users-list), [hot keys](https://github.com/AAndyProgram/SCrawler/wiki#hot-keys))
|
|
2. Click the `Download` button, then `Standalone downloader` ([read more here](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video))
|
|
|
|
|
|
|
|
# Contact me
|
|
|
|
[e-mail](mailto:andyprogram@proton.me): andyprogram@proton.me
|
|
|
|
Matrix (Element): https://matrix.to/#/@andyprogram:matrix.org
|
|
|
|
Discord (contact the developer): andyprogram
|
|
|
|
Discord server: https://discord.gg/uFNUXvFFmg
|
|
|
|
[Wire](https://account.wire.com/user-profile/?id=93985052-cf2c-4b72-ac75-bbe3231cf544): @andyprogram
|