geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2024.2.25.0

Andy
2024-02-25 11:33:42 +03:00
parent a137922daa
commit d2ce5e2232
3 changed files with 109 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
Home.md

@@ -105,8 +105,12 @@ Context menu:
- `Open user` (with folder image) - open user folder
- `Open user` (with globe) - open user site
- `Open post` - open post in browser
- `Copy to...` - copy item to another location
- `Move to...` - move item to another location
- `Add to Favorite` - this post will be added to your favorite feed
- `Add to Favorite (remove from current)` - this post will be added to your favorite feed and removed from the current feed/session
- `Add to special feed...`- this post will be added to the feed(s) you select (a selection form will open)
- `Add to special feed (remove from current)...`- this post will be added to the feed(s) you select (a selection form will open) and removed from the current feed/session
- `Remove from Favorite` - this post will be removed from your favorite feed
- `Remove from special feed...` - this post will be removed from the feed(s) you select (a selection form will open)
- `Find user` - find user in the main window
@@ -119,11 +123,15 @@ Load sessions:
- `Load current session`
- `Load last session` - load last saved session data
- `Select loading session` - load the data of the session you have chosen (the session selection form will open)
- `Copy checked to...` - copy checked items to another location
- `Move checked to...` - move checked items to another location
- `Load Favorite` - load your Favorite feed
- `Load special feed` - load one or more of your special feeds
- `Add checked to Favorite` - add checked posts to your Favorite feed
- `Add checked to Favorite (remove from current)` - add checked posts to your Favorite feed and remove them from the current feed/session
- `Remove checked from Favorite` - remove checked posts from your Favorite feed
- `Add checked to special feed...` - add checked posts to one or more of your feeds
- `Add checked to special feed (remove from current)...` - add checked posts to one or more of your feeds and remove them from the current feed/session
- `Remove checked from special feed...` - remove checked posts from one or more of your feeds
- `Clear Favorite` - remove all posts from your Favorite feed (just clearing the feed; **files will not be deleted**)
- `Clear special feed...` - remove all posts from one or more of your feeds (just clearing the feeds; **files will not be deleted**)
@@ -132,17 +140,27 @@ Load sessions:
- `Delete daily feed (by date)` - delete session feeds by the date (the date selection form will open; **files will not be deleted**)
- `Merge sessions` - merge multiple session feeds into one
- `Clear session` - clear list of recent downloads
- `Merge special feeds` - merge multiple special feeds into one
- `Select all/none`
Change view:
- `Downloads` - show downloaded regular users
- `Subscriptions` - show downloaded [subscriptions](#subscriptions)
**Move/Copy media:**
- `Destination` - The destination folder for the media you want to move/copy. *If checked, the profile path will be ignored when changing profiles.*
- `Place videos in a separate video folder` - The videos will be placed in a separate folder (`Video`) at the destination.
- `Replace user profile` - The user profile will be replaced with the selected one. This may be suitable for cases where you're deleting a user profile. Feeds' media are associated with the user's profile. If you delete a profile, all of that profile's media will disappear from all of your feeds. If you change the user profile, this won't happen! **Only works when moving media!**
- `Create path profile` - Create a path profile if it doesn't exist. This means that if you checked `Replace user profile` and didn't select a replacement profile, a new `Path` profile (path plugin) will be created and added to SCrawler. **Only works when moving media!**
- `Profile` - List of user profiles (without path plugin). **Only works when moving media!**
- `Profile (path)` - List of user profiles (path plugin only). **Only works when moving media!**
![Feed move copy media](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/FeedMoveCopyTo.png)
Hot-keys:
- `F5` - refresh
- `Ctrl+G` - go to a specific page
*The buttons are only displayed if the `Store session data` option is [enabled](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed)*
![Feed session context menu](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/FeedWindowSessionContext.png)
Hot keys
@@ -210,7 +228,9 @@ This allows you to pause the automation for a while. The pause will remain in ef
![Pause context](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindowPause.png)
### View
- `View` - users view modes.
Users view modes.
- The first block specifies **how** users will be displayed in the form.
- `Large/Small images` - with icons.
- `List` / `Details` - simple, without icons.
@@ -237,6 +257,8 @@ This allows you to pause the automation for a while. The pause will remain in ef
- `Limit dates (not in range)` - filter users whose last download date is not in the selected date range
- `Limit dates (in range)` - filter users whose last download date is in the selected date range
**You can also change the view by (un)grouping by site using the `Show groups` and `Use user grouping` [options in the settings](https://github.com/AAndyProgram/SCrawler/wiki/Settings#basis).**
![Main window view context](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/MainWindowView.png)
### LOG
@@ -281,6 +303,8 @@ Double-click the user to open the content folder.
## Download separate video
`Download` - `Standalone downloader`
Read [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader)
![Separate video downloader window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/AppYouTube.png)
@@ -291,6 +315,7 @@ Read [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader)
- [`Channels`](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
- [`Feed`](#feed)
- [`Downloader`](#download-separate-video)
- [`Scheduler`](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) - show scheduler
- `Show/Hide` - show/hide SCrawler
- `Close` - close the program
- `Close (no script)` - close the program ignore the script (if you configured the script in the settings)
@@ -313,6 +338,9 @@ Paste the copied URL into the `User name` field. **Ignore if the username is set
If you want to add a folder as a user, you need to use the following pattern: `D:\Data\`.
**ATTENTION!**
<br/>**ALWAYS PASTE THE USER PROFILE URL**
![Add user](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/CreateUserClear.png)
- `User name` - **Paste the user profile URL here.**
@@ -510,5 +538,3 @@ The program implements two ways to add users to this list:
1. Check the `SpecialPath` values to make sure they still point to the correct locations. If not, replace these values with the correct ones. To find all non-empty `SpecialPath` values, you can use a simple regular expression: `SpecialPath="[^"]+"`.
1. Save this file and close.
1. Open SCrawler.

4
Plugins.md

@@ -545,6 +545,10 @@ Assembly attribute. Not used yet, but in future versions will allow SCrawler to
- `Name` - your GitHub username
- `RepoName` - your repository name
## ReplaceInternalPluginAttribute
If you want your plugin to overwrite the internal one, use this attribute.
# PropertyValue value types
| vb.net | C# | NET |

52
Settings.md

@@ -126,6 +126,7 @@
- `Open last mode (users or subscriptions)` - if disabled, the user mode will be used when initializing the feed.
- `Show friendly names instead of usernames`
- `Show special feeds in media items` - if you have one or more special feeds, they will appear in the additional post context menu. Disabled by default. **Enabling this setting may increase memory consumption.**
- `Update file location when moved` - the file location will be updated in the session data and in the feeds data.
![Feed settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsGlobalFeed.png)
@@ -172,24 +173,47 @@ Red highlight means that the field is required or you have entered an incorrect
- `Get user media only` - the default for creating new users
- `Temporary`, `Download images/videos` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the `Defaults` tab. If the checkbox is in an intermediate state, then the same name setting of the `Defaults` tab will be used instead. Otherwise, this setting will be applied.
## ffmpeg
## Environment
### ffmpeg
ffmpeg is required for several sites like Reddit, PornHub, XHamster, XVIDEOS, etc. **It is included in the SCrawler release, but you can download ffmpeg from:**
- x64 version - [release](https://github.com/GyanD/codexffmpeg/releases/tag/5.1.2); [zip](https://github.com/GyanD/codexffmpeg/releases/download/5.1.2/ffmpeg-5.1.2-full_build.zip); **version `5.1.2-full_build-www.gyan.dev`**
- x86 version - [release](https://github.com/yt-dlp/FFmpeg-Builds/releases/tag/autobuild-2022-11-30-12-57); [zip](https://github.com/yt-dlp/FFmpeg-Builds/releases/download/autobuild-2022-11-30-12-57/ffmpeg-N-109274-gd7a5f068c2-win32-gpl.zip); **version `N-109457-geeb280f351-20221226`**
## Gallery-dl
### Gallery-dl
Version **1.26.4** ([release](https://github.com/mikf/gallery-dl/releases/tag/v1.26.4); [exe](https://github.com/mikf/gallery-dl/releases/download/v1.26.4/gallery-dl.exe))
## YT-DLP
**The following sites use gallery-dl:**
- [Pinterest](#pinterest)
- [Twitter](#twitter)
### YT-DLP
Version **2023.11.16** ([release](https://github.com/yt-dlp/yt-dlp/releases/tag/2023.11.16))
- x64 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2023.11.16/yt-dlp.exe)
- x86 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2023.11.16/yt-dlp_x86.exe)
**The following sites use yt-dlp:**
- [YouTube](#youtube)
- [ThisVid](#thisvid)
- [TikTok](#tiktok) (using [plugin](https://github.com/bashonly/yt-dlp-TTUser))
### OF-Scraper
Version **2023.12.25_485586_main_d584b2b_package** ([release](https://github.com/datawhores/OF-Scraper/releases/tag/d584b2b))
**Don't download the `exe` file. Always download `zip`.**
**Don't put the program in the SCrawler's `Environment` folder because if you use the updater to update SCrawer, you'll lose 'OF-Scraper'. I recommend creating an `Environment2` folder and placing 'OF-Scraper' there.**
**Don't rush to update 'OF-Scraper'. Users using 'OF-Scraper' report that minor releases contain many bugs.**
*Go to the [OnlyFans OF-Scraper chapter](#of-scraper-support).*
## User query options
**Available for search queries only. Not used when creating users.**
@@ -299,6 +323,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
**To download DRM protected videos, you need to download [OF-Scraper](https://github.com/datawhores/OF-Scraper) and add it to the site settings in SCrawler.**
**Don't forget to read [THIS](#of-scraper)!**
1. Go to the [latest release](https://github.com/datawhores/OF-Scraper/releases/latest)
1. Download zip file for Windows
1. Extract it to any folder you want
@@ -338,11 +364,12 @@ To support the downloading of this site you should add the Mastodon domain to th
## Instagram
- Authorization
- `x-csrftoken` - [how to find](#how-to-find-headers)
- `x-csrftoken` - [how to find](#how-to-find-headers) *(can be automatically extracted from cookies)*
- `x-ig-app-id`, `ix-ig-www-claim`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-headers)
- `Saved posts user` - your personal Instagram username to download your saved posts
- Other parameters
- `Download timeline` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download reels` - Download reels *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download stories` - Download stories *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download stories: user` - Download active (non-pinned) user stories *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
- `Download tagged` - Download tagged posts *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
@@ -350,6 +377,7 @@ To support the downloading of this site you should add the Mastodon domain to th
- `Request time counter` - how many requests will be sent to Instagram before the program waits `Request timer` milliseconds (**it is highly recommended not to change this default value**)
- `Post limit timer` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
- `Get timeline` - default value for new users
- `Get reels` - default value for new users
- `Get stories` - default value for new users
- `Get stories: user` - default value for new users
- `Get tagged photos` - default value for new users
@@ -359,6 +387,15 @@ To support the downloading of this site you should add the Mastodon domain to th
![Instagram settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteInstagram.png)
### Instagram data means
1. Avatar *(download only for the first time)*
2. `Reels`
3. `Tagged posts`
4. `Stories`
5. `Stories: user`
![Instagram site blocks](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteInstagramData.png)
### Instagram requirements
- **[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY**
- [`x-csrftoken`](#how-to-find-headers)
@@ -459,6 +496,8 @@ result_cancel-->[*]
The `Accept` and `UserAgent` headers must be obtained from the page `https://facebook.com/<USERNAME>`.
If you can't find `x-ig-app-id` or `x-asbd-id`, you can get them from Instagram.
![Facebook settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteFacebook.png)
### Facebook requirements
@@ -506,6 +545,7 @@ The `User ID` and `UserAgent` values must be obtained from the page `https://jus
- `Download user playlists`
- `Download user community: images`
- `Download user community: videos`
- `Ignore community errors` - if true, community errors will not be added to the log.
- `Use cookies` - default value for new users. Use cookies to download the user.
See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader#Settings).
@@ -532,6 +572,8 @@ See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/You
- `Use native title` - use a user-created video title for the filename instead of the video ID
- `Use native title in standalone downloader` - use a user-created video title for the filename instead of the video ID
- `Add video ID to video title` - the video ID will be added to the file name
- `Use regex to clean video title`
- `Title regex` - regex to clean video title (only works with checked `Use regex to clean video title`)
- `Use video date as file date` - set the file date to the date the video was added (website) (if available)
![TikTok settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteTikTok.png)
@@ -725,6 +767,8 @@ This command can be a batch command or any script file (bat, ps1 or whatever you
1. Scroll down to `Request Headers`.
1. Find the headers you need, copy and paste them into the corresponding fields.
**You can also use `Ctrl+F` to find the header you need**
# How to find Imgur client ID
1. Open browser developer tools
1. Go to `Network`