SCrawler/README.md

🏳️‍🌈 Social networks crawler 🏳️‍🌈

🇪🇺 🇬🇷

A program to download photo and video from any site (e.g. YouTube, YouTube Music, Reddit, Twitter, Mastodon, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, ThisVid, LPSG, Pinterest).

If you like SCrawler, please like the program on this site and/or this

Bitcoin: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET

YouTube standalone application:

What can program do:

• Download pictures and videos from users' profiles and subreddits:
  • YouTube videos, shorts, users, artists, playlists, music, tracks;
  • Reddit images, galleries of images, videos, saved posts;
  • Redgifs videos (https://www.redgifs.com/);
  • Twitter images and videos, saved (bookmarked) posts;
  • Mastodon images and videos, saved (bookmarked) posts;
  • Instagram images and videos, tagged posts, stories, saved posts;
  • TikTok videos (currently broken; limited);
  • Pinterest boards, users, saved posts;
  • Imgur images, galleries and videos;
  • Gfycat videos;
  • PornHub images, videos, save (liked) posts;
  • XHamster images, videos, saved posts;
  • XVIDEOS videos, saved posts;
  • ThisVid images, videos, saved posts;
  • Other supported sites
• Parse channel and view data
• Download saved Reddit, Twitter and Instagram posts
• Add users from parsed channel
• Advanced user management
• Automation (downloading data automatically every X minutes)
• Feed (feed of downloaded media files)
• Labeling users
• Create download groups
• Adding users to favorites and temporary
• Filter exists users by label or group
• Selection of media types you want to download (images only, videos only, both)
• Download a special video, image or gallery
• Making collections (grouping users into collections)
• Specifying a user folder (for downloading data to another location)
• Changing user icons
• Changing view modes
• ...and many others...

Supported sites

• YouTube
• YouTube Music
• Reddit
• Twitter
• Mastodon
• Instagram
• TikTok (currently broken; limited)
• RedGifs
• Pinterest
• Imgur
• Gfycat
• LPSG
• PornHub
• XHamster
• XVIDEOS
• ThisVid
• Other sites

SITES REQUIREMENTS

How it works

First, the program downloads the full profile. After the program downloads only new posts. The program remembers downloaded posts.

Reddit

The program parses user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.

Other sites

The program parses user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.

How to request a new site

I'm currently not accepting requests to develop new sites.

Requirements

• Windows 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this tool.
• SITES REQUIREMENTS

Guide

• Main window
  • Users
    • Add/Edit/Delete users
    • Collections
    • User operations
    • User labels
  • DOWNLOAD
    • Automation
    • Download groups
  • Downloading information
  • Reddit channels
  • Saved posts
  • View modes, filters
• SETTINGS
  • SITES REQUIREMENTS
  • Reddit
  • Twitter
  • Mastodon
  • Instagram
  • TikTok
  • RedGifs
  • YouTube
  • Pinterest
  • PornHub
  • XHamster
  • XVIDEOS
  • ThisVid
  • LPSG

Full guide you can find here

Installation

Just download the latest release, unzip the program archive to any folder and enjoy. 😊

Don't put program in the Program Files system folder (this is portable program and program settings are stored in the program folder)

Updating

Just download latest version and unpack it into the program folder. Before starting a new version, I recommend making a backup copy of the program settings folder.

How to build from source

How to make a plugin

How to support

Settings and usage

The program has an intuitive interface.

SITES REQUIREMENTS

Just add a user profile and click the Download button.

stateDiagram
Start: Add site credentials
What: What would I like to do
DownUser: Download user
DownVideo: Download video
AUser: Add user (1)
OVIF: Open standalone downloader (2)
AVideo: Add video url
F5: Press 'F5' or click the download button
[*]-->Start
Start-->What
What-->DownUser
What-->DownVideo
DownUser-->AUser
DownVideo-->OVIF
OVIF-->AVideo
AVideo-->F5
AUser-->F5
F5-->[*]

1. Press Insert or click the Download button (read more here, hot keys)
2. Click the Download button, then Standalone downloader (read more here)

Contact me

Matrix (Element): https://matrix.to/#/@andyprogram:matrix.org

Discord: AndyProgram#3804