geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2022.10.18.0

Andy
2022-10-18 12:38:15 +03:00
parent d4faaa1cfa
commit 7a464f1c23
4 changed files with 282 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files

167
Home.md

@@ -6,8 +6,14 @@ This window contains all data management tools.
- ```Add/Edit/Delete``` - actions with users.
- ```Refresh``` - refresh the user list. You do not need to press this button. Added just in case...
- ```Info``` - show downloaded users and what exactly and how many were downloaded.
- ```Feed``` - show recently downloaded data
- ```Info``` - [show](#info) downloaded users and what exactly and how many were downloaded.
- ```Feed``` - [show](#feed) recently downloaded data
- ```Channels``` - open [Channels](https://github.com/AAndyProgram/SCrawler/wiki/Channels) form
- ```Saved posts``` - download [saved posts](#saved-posts)
- ```Download``` - [download](#download) operations
- ```Stop``` - stop downloading
- ```View``` - [change](#view) view mode
- ```LOG``` - program error log and information
## Buttons
@@ -15,7 +21,7 @@ This window contains all data management tools.
This form will show downloaded items (sorted by latest download date). Each line of data contains information about how many images and videos have been downloaded for the user. If no media is downloaded, the user will not appear in this form.
*Right-clicking on this button opens the 'Missing posts' window.*
*Right-clicking on this button opens the '[Missing posts](#missing-posts)' window.*
Double-clicking on the data line will open the content folder.
@@ -37,39 +43,8 @@ Hot keys:
![Info window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/DownloadInfo.png)
### Feed
This form will show the recently downloaded data (images and videos). Each image/video contains download information (tooltip), file address, user information, etc.
Double-clicking on an image will open the original image in an external viewer.
Buttons:
- ```Refresh``` - the feed will be updated to receive new downloaded items and/or remove missing items
- ```Clear``` - clear list of recent downloads
- ```Delete selected``` - delete marked items from the disk
Context menu:
- ```Open picture/video``` - open picture/video in external viewer
- ```Open user``` (with folder image) - open user folder
- ```Open user``` (with globe) - open user site
- ```Open post``` - open post in browser
- ```Find user``` - find user in the main window
- ```Information``` - show post information
- ```Delete picture/video``` - delete picture/video
Hot keys
- ```F5``` - ```Refresh```
- ```F3``` - Previous page
- ```F4``` - Next page
You can read about the settings [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed).
![Feed window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindow.png)
### Missing posts
**Still under development**
This form shows posts that have not been downloaded. This form is opened by right-clicking on the '[Info](#info)' button in the main window
Buttons:
@@ -87,10 +62,49 @@ Context menu:
- ```Delete post``` - delete selected post
Hot keys:
- ```F1``` - ```Update```
- ```F1``` - ```Info```
![Missing posts window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MissingPosts.png)
### Feed
This form will show the recently downloaded data (images and videos). Each image/video contains download information (tooltip), file address, user information, etc.
Double-clicking on an image will open the original image in an external viewer.
![Feed window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindow.png)
Buttons:
- ```Refresh``` - the feed will be updated to receive new downloaded items and/or remove missing items
- ```Clear``` - clear list of recent downloads
- ```Delete selected``` - delete marked items from the disk
Context menu:
- ```Open picture/video``` - open picture/video in external viewer
- ```Open user``` (with folder image) - open user folder
- ```Open user``` (with globe) - open user site
- ```Open post``` - open post in browser
- ```Find user``` - find user in the main window
- ```Information``` - show post information
- ```Delete picture/video``` - delete picture/video
![Feed context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindowItemContext.png)
Load sessions:
- ```Load last session``` - load last saved session data
- ```Select loading session``` - Load the data of the session you have chosen (the session selection form will open)
*The buttons are only displayed if the ```Store session data``` option is [enabled](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed)*
![Feed session context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/FeedWindowSessionContext.png)
Hot keys
- ```F5``` - ```Refresh```
- ```F3``` - Previous page
- ```F4``` - Next page
You can read about the settings [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#feed).
### Channels
The channels form will open.
@@ -99,34 +113,49 @@ More about channels [here](https://github.com/AAndyProgram/SCrawler/wiki/Channel
### Saved posts
**This function requires [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
**[Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash) and cookies required to download saved Instagram posts**
Go to Settings - Settings - Reddit/Instagram/Twitter. Enter your username in the "Saved posts user" textbox. Click OK. Go to the main window.
Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the ```Saved posts user``` textbox. Click OK. Go to the main window.
This button looks like a bookmark. When you click on this button, a form for downloading saved posts will open. Saved posts are stored in the Reddit/Instagram/Twitter data path in the ```!Saved posts``` folder.
![Saved posts window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SavedPosts.png)
### Download
- ```Download selected``` - start downloading one or more selected users.
- ```Download all```: **ATTENTION! THE BUTTONS IN THIS MENU DO NOT RESPECT VIEW MODES (SUCH AS LABELS AND/OR GROUPS), FILTERS, AND OTHER OPTIONS.!**
- ```Download all``` - Download all users marked ```Ready for download``` from all sites.
- ```Download selected (F5)``` - start downloading one or more selected users.
- ```Download all```: **ATTENTION! THE BUTTONS IN THIS MENU DO NOT RESPECT VIEW MODES (SUCH AS LABELS AND/OR GROUPS), FILTERS, AND OTHER VIEW OPTIONS.!**
- ```Download all (F6)``` - Download all users marked ```Ready for download``` from all sites.
- ```Download all site users``` - Download all users marked ```Ready for download``` from specific sites.
- ```Download all FULL``` - Download all users from all sites. **The ```Ready for download``` option will be ignored.**
- ```Download all site users FULL``` - Download all users from specific sites. **The ```Ready for download``` option will be ignored.**
- ```Add a new download group``` - Create a new download group (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups))
- ```Automation``` - Automatic download settings (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)
- ```Pause automation``` - Pause the automation process. This allows you to pause the automation for a while. The pause will remain in effect until you press this button again or restart SCrawler.
- ```Download video``` - download a separate video (Reddit and Twitter videos supported) or Instagram post (photo and video).
- ```Standalone downloader``` - Just a [downloader](#download-separate-video) for single videos.
- ```Add a new download group``` - Create a new download group (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)).
- ```Silent mode``` - Temporarily disabling notifications.
- ```Automation``` - Automatic download settings (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)).
- ```Pause automation``` - [Pause the automation process](#pause-automation).
- ```Stop``` - stop all download operations.
![Saved posts window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
[HotKeys](#hot-keys)
![Download context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
#### Pause automation
This allows you to pause the automation for a while. The pause will remain in effect until you press this button again or restart SCrawler.
- ```1 hour``` - pause automation for 1 hour
- ```2 hour``` - pause automation for 2 hours
- ```3 hour``` - pause automation for 3 hours
- ```4 hour``` - pause automation for 4 hours
- ```6 hour``` - pause automation for 6 hours
- ```12 hour``` - pause automation for 12 hours
- ```Until``` - you will be prompted to enter the time you want to pause the task(s)
- ```Unlimited``` - pause the task until you turn it off or close the program
- ```Disable``` - disable pause
![Pause context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowPause.png)
### View
- ```View``` - users view modes.
- The first block specifies **how** users will be displayed in the form.
- ```Large/Small images``` - with icons.
@@ -150,20 +179,22 @@ This button looks like a bookmark. When you click on this button, a form for dow
- ```Limit dates (not in range)``` - filter users whose last download date is not in the selected date range
- ```Limit dates (in range)``` - filter users whose last download date is in the selected date range
![Main window context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindow2.png)
![Main window view context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowView.png)
### LOG
The error log will be opened by clicking on this button. The log contains program and download errors. The log is saved automatically when you close the program, but you still can save, copy, or clear it (using the special buttons on the log form) if you want...
## Hot keys
- ```F1``` - program information and checking for a new version
- ```Insert``` - add user
- ```F3``` - edit user
- ```F5``` - start downloading one or more selected users
- ```F6``` - start downloading all your users marked as ```Ready for download```.
- ```F2``` - download a separate video or Instagram post
- ```Del``` - delete one or more selected users
- ```Enter``` - open user folder
- ```F3``` - edit user
- ```F5```, ```Ctrl+F5``` - start downloading one or more selected users (pressing with ```Ctrl``` excludes this data from the feed)
- ```F6```, ```Ctrl+F6``` - start downloading all your users marked as ```Ready for download``` (pressing with ```Ctrl``` excludes this data from the feed)
- ```Ctrl+F``` - open search form
- ```Ctrl+NumericKey``` - download group with this number
Double-click the user to open the content folder.
@@ -179,6 +210,22 @@ Double-click the user to open the content folder.
![Separate video downloader window](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SeparateVideoDownloader.png)
## Tray
- ```Silent mode``` - Temporarily disabling notifications.
- ```Pause automation``` - [Pause the automation process](#pause-automation).
- [```Feed```](#feed)
- ```Show/Hide``` - show/hide SCrawler
- ```Close``` - close the program
- ```Close (no script)``` - close the program ignore the script (if you configured the script in the settings)
![Tray context menu](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/TrayContextMenu.png)
## Search users
```Ctrl+F``` to start searching
![Search form](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SearchUsers.png)
## Users list
### Context menu
@@ -217,15 +264,11 @@ When you add a user to a collection, the user's data will be moved to the collec
When you add a user to the collection, a new folder will be created in the collection folder named by pattern ```Site```_```UserName```. That is, each profile in the collection is stored separately. When you merge, all files in the profiles' folders are moved to the root of the collection.
### Deleting a collection
1. Select the collection you want to delete.
1. Click the ```Delete``` button or right-click on the collection and select ```Delete user / collection```.
1. In the dialog box, click ```Delete``` button.
- You will be prompted to delete the collection and all of its files:
- ```Yes``` - The collection and all its files will be permanently deleted.
- ```No``` - You will be prompted to split the collection.
- ```Yes``` - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted.
- ```No``` - Cancel operation.
- ```Delete``` - The collection and all its files will be permanently deleted.
- ```Split``` - The collection will be split: each user profile in the collection will be displayed as a separated profile (as is), and the collection will be deleted.
- ```Cancel``` - Cancel operation.
### Collections example

13
Plugins.md

@@ -128,7 +128,7 @@ interface ISiteSettings
string GetUserUrl(string UserName,bool Channel);
ExchangeOptions IsMyUser(string UserURL);
ExchangeOptions IsMyImageVideo(string URL);
IEnumerable<PluginUserMedia> GetSpecialData(string URL);
IEnumerable GetSpecialData(string URL);
IPluginContentProvider GetInstance(Download What);
void Load(IEnumerable<KeyValuePair<string, string>> XMLValues);
IEnumerable<KeyValuePair<string, string>> XMLFields();
@@ -140,6 +140,8 @@ interface ISiteSettings
void BeforeStartDownload(object User, Download What);
void AfterDownload(object User, Download What);
void DownloadDone(Download What);
void BeginEdit();
void EndEdit();
void Update();
void Reset();
void OpenSettingsForm();
@@ -229,7 +231,7 @@ Must return the same as the [```IsMyUser```](#ismyuser), but ```UserName``` can
### GetSpecialData
```Function GetSpecialData(ByVal URL As String) As IEnumerable(Of PluginUserMedia)```
```Function GetSpecialData(ByVal URL As String) As IEnumerable```
After ```IsMyImageVideo```, if this is your plugin's media file, this function will be called.
@@ -275,6 +277,13 @@ Called at the start of host initialization.
### EndInit
Called at the end of host initialization.
### BeginEdit
Called when opening the settings form in SCawler
### EndEdit
Called when closing the settings form in SCawler
### BeginUpdate

173
Settings.md

@@ -14,7 +14,6 @@
![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBasis.png)
## Behavior
- ```Exit confirm``` - ask for confirmation before closing the program
- ```Close to tray``` - close program to system tray
- ```Show notifications``` - show notifications when download is complete
@@ -26,7 +25,17 @@
- ```Folder cmd``` - the [command](#folder-command) to open a folder
- ```Close cmd``` - this [command](#scrawler-script-text-examples) will be executed when SCrawler is closed
![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBehavior.png)
![Behavior settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBehavior.png)
## Notifications
- ```Silent mode``` - Temporarily disabling notifications. This setting is not stored in the settings file. It is valid until you turn it off or close the program.
- ```Show notification``` - This is the base value of notifications. If you disable it, notifications will not appear at all.
- ```Profiles``` - Show notifications when profiles download is complete.
- ```AutoDownloader``` - Show AutoDownloader notifications.
- ```Channels``` - Show notifications when channels download is complete.
- ```Saved posts``` - Show notifications when saved posts download is complete.
![Notifications settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalNotifications.png)
## Defaults
- ```Separate video folder``` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images.
@@ -50,6 +59,7 @@
- ```After download cmd``` - this [command](#scrawler-script-text-examples) will be executed after all downloads are completed
- ```Add missing information to log``` - Text will be added to the log stating that the [missing posts](https://github.com/AAndyProgram/SCrawler/wiki/#missing-posts) exist.
- ```Add missing errors to log``` - Each error that prevents SCrawler from downloading a file will be added to the log.
- ```Trying to download missing posts using regular download``` - If missing posts exist, the missing posts will attempt to be downloaded via user download.
![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDownloading.png)
@@ -62,56 +72,78 @@
![Channels settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalChannels.png)
## Feed
- ```Feed rows/columns``` - how many rows and columns will be displayed in the feed form
- ```Endless feed``` - go to the next page when you reach the end
- ```Add the session number to the post title``` - The session number will be displayed to the left of the post title. A session is every time when you (or [AutoDownloader](#automation)) download data.
- ```Add the date to the post title``` - The download date and time will be displayed to the right of the post title.
- ```Store session data``` - If checked, session data will be stored in an xml file.
![Feed settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalFeed.png)
# Site settings
## Reddit
## Sites requirements
The Reddit parser can parse data without cookies, but you can add it if you like.
- [Reddit](#reddit-requirements)
- [Twitter](#twitter-requirements)
- [Instagram](#instagram-requirements)
- [RedGifs](#reddit-requirements)
- [TikTok](#tiktok-requirements)
- [XVIDEOS](#xvideos-requirements)
- [LPSG](#lpsg-requirements)
## Site default fields
- ```Path``` - where Reddit users' data will be placed.
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. Saved Twitter posts are posts that you have bookmarked.
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
- ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
- ```Get user media only``` - the default for creating new users
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
## ffmpeg
To download videos hosted on Reddit you need to download the ffmpeg.exe and place it in the SCrawler folder. It is also required for xvideos downloader. **You can download ffmpeg from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip)**
**ffmpeg only works with the x64 version of the program.**
## Reddit
The Reddit parser can parse data without cookies, but you can add it if you like.
- ```Use M3U8``` - If you are using an x86 program, you will not be able to use the M3U8 download. In other cases, this setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteReddit.png)
### Reddit user settings
### Reddit requirements
- [Cookies](#how-to-set-up-cookies) for downloading saved posts
- [ffmpeg](#ffmpeg) for downloading videos hosted on reddit (only for x64 program)
- [Imgur clien ID](#how-to-find-imgur-client-id) to download Imgur content posted on reddit
- [RedGifs credentials](#redgifs-requirements) to download RedGifs content posted on reddit
### Reddit user settings
- ```View``` - works the same as Reddit view modes (```new```, ```hot```, ```top```)
- ```Period``` - only works with ```Top``` view mode and the same as Reddit periods
![Reddit user settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsRedditView.png)
## Twitter
- ```Path``` - where Twitter users' data will be placed.
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. Saved Twitter posts are posts that you have bookmarked.
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
- ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
- ```Get user media only``` - the default for creating new users
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space.
- ```Token``` - in this field you need to put ```x-csrf-token```.
- [```Authorization```](#how-to-find-twitter-tokens) - Bearer token. Must start with ```Bearer ``` word followed by a space.
- [```Token```](#how-to-find-twitter-tokens) - in this field you need to put ```x-csrf-token```.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (bookmarked).
![Twitter settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteTwitter.png)
### How to find Twitter tokens
### Twitter requirements
- [```Cookies```](#how-to-set-up-cookies)
- [```Authorization```](#how-to-find-twitter-tokens)
- [```Token```](#how-to-find-twitter-tokens)
- ```Saved posts user``` for download saved posts (bookmarked)
### How to find Twitter tokens
1. Open Google Chrome, Edge or FireFox.
1. Go to https://twitter.com
1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl+Shift+I```).
1. Go to the network tab
1. In the window where you have twitter open, click on first post you see (**CLICK ON THE POST DATE**)
1. Then look at the dev tools and in the first lines you will see what you need! In the list on the left, find a query that starts with ```TweetDetail?```... and click on it
@@ -121,19 +153,12 @@ The Reddit parser can parse data without cookies, but you can add it if you like
## Instagram
- ```Path``` - where Instagram users' data will be placed.
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
- ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
- ```Get user media only``` - the default for creating new users
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- Authorization
- ```Hash``` - **\[Required\]** in this field you need to put the hash of the Instagram session ([how to find](#how-to-find-instagram-hash)).
- ```Hash 2``` - **\[For saved Instagram posts only\]** in this field you need to put the [hash of the Instagram session for saved posts](#how-to-find-instagram-hash).
- ```x-csrftoken``` - **\[Required\]** read [here](#how-to-find-instagram-authorization-headers) how to find them
- ```x-ig-app-id```, ```ix-ig-www-claim``` - **\[Required\]** read [here](#how-to-find-instagram-authorization-headers) how to find them
- ```Saved posts user``` - your personal Instagram username to download your saved posts (this feature requires cookies and **InstaHash 2**)
- ```Hash``` - in this field you need to put the hash of the Instagram session ([how to find](#how-to-find-instagram-hash)).
- ```Hash 2``` - in this field you need to put the [hash of the Instagram session for saved posts](#how-to-find-instagram-hash).
- ```x-csrftoken``` - [how to find](#how-to-find-instagram-authorization-headers)
- ```x-ig-app-id```, ```ix-ig-www-claim``` - [how to find](#how-to-find-instagram-authorization-headers)
- ```Saved posts user``` - your personal Instagram username to download your saved posts
- Other parameters
- ```Request timer``` - this is the time value (in milliseconds) the program will wait before processing the next ```Request time counter``` request (**it is highly recommended not to change this default value**)
- ```Request time counter``` - how many requests will be sent to Instagram before the program waits ```Request timer``` milliseconds (**it is highly recommended not to change this default value**)
@@ -144,8 +169,39 @@ The Reddit parser can parse data without cookies, but you can add it if you like
![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteInstagram.png)
### How to find Instagram Hash
- Other settings
- Download Timeline
- Download Stories and Tagged data
- Download saved posts
**ATTENTION!**
**These settings are not for configuration! Sometimes SCrawler determines some of the error codes as expired credentials. In order to protect your Instagram account from being banned, SCrawler temporarily disables the download module where an error occurred. In this case, a notification will be added to the log. If you have verified your credentials and there are no changes, you can simply re-enable the module manually.**
**When you change credentials, SCrawler changes these values depending on the required fields.**
![Instagram additional settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteInstagramAdditional.png)
### Instagram requirements
**[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY**
- To download timeline, stories and tagged posts
- [```Cookies```](#how-to-set-up-cookies)
- [```Hash```](#how-to-find-instagram-hash)
- [```x-csrftoken```](#how-to-find-instagram-authorization-headers)
- [```x-ig-app-id```](#how-to-find-instagram-authorization-headers)
- [```ix-ig-www-claim```](#how-to-find-instagram-authorization-headers)
- To download saved posts
- [```Cookies```](#how-to-set-up-cookies)
- [```Hash 2```](#how-to-find-instagram-hash)
- [```x-csrftoken```](#how-to-find-instagram-authorization-headers)
- [```x-ig-app-id```](#how-to-find-instagram-authorization-headers)
- [```ix-ig-www-claim```](#how-to-find-instagram-authorization-headers)
- ```Saved posts user```
### How to find Instagram Hash
1. Open browser developer tools
1. Go to ```Network```
1. If this tab contain any data, clear it.
@@ -157,12 +213,11 @@ The Reddit parser can parse data without cookies, but you can add it if you like
1. Copy this and paste it into the corresponding (```Hash```, ```Hash 2```) field on the Instagram settings form.
### How to find Instagram authorization headers
1. Open browser developer tools
1. Go to ```Network```
1. If this tab contain any data, clear it.
1. Open profile tagged photos.
1. Click ```Ctrl``` + ```F``` and find the text ```x-ig-app-id``` and ```x-ig-www-claim```.
1. Click ```Ctrl+F``` and find the text ```x-ig-app-id``` and ```x-ig-www-claim```.
1. Copy the values and paste them into the corresponding fields on the Instagram settings form.
### Instagram limits
@@ -186,16 +241,49 @@ When the number of tagged posts exceeds the tagged posts limit, you will be aske
- ```Cancel``` - Cancel tagged posts download operation
## RedGifs
- ```Path``` - where RedGifs users' data will be placed.
- ```Saved posts path``` - *unused*
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
- ```Download site data``` - You can disable downloading data from the site if you need it. If disabled, this site's data will not be downloaded.
- ```Get user media only``` - the default for creating new users
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- ```Token``` - "Bearer" token ([how to find](#how-to-find-redgifs-token))
![RedGifs settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteRedGifs.png)
### RedGifs requirements
- [```Cookies```](#how-to-set-up-cookies)
- [```Token```](#how-to-find-redgifs-token)
### How to find RedGifs token
1. Open browser developer tools
1. Go to ```Network```
1. If this tab contain any data, clear it.
1. Open any RedGifs profile.
1. Click ```Ctrl+F``` and find the text ```Bearer```.
1. Copy the value and paste it into the ```Token``` field on the RedGifs settings form.
## TikTok
![TikTok settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteTikTok.png)
### TikTok requirements
- [```Cookies```](#how-to-set-up-cookies)
### TikTok limits
Unfortunately, I didn't understand how to use the 'Cursor' option to get the JSON response. In this case, I can only parse an HTML page that only provides the latest 30 videos. **If you know how to use the cursor option to get a JSON response, you might create an issue with your suggestion.**
## XVIDEOS
- ```Download UHD``` - Download UHD (4K) content
![XVIDEOS settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteXvideos.png)
### XVIDEOS requirements
Nothing is needed to download xvideos content, not even cookies.
## LPSG
![LPSG settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteLPSG.png)
### LPSG requirements
- [```Cookies```](#how-to-set-up-cookies)
# Download groups
In many cases, you may need to download some users. You can group these users and make it easier to download them without having to select them every time.
@@ -258,7 +346,6 @@ You can specify a command to open a folder in a special program. Pattern: ```Com
This command can be a batch command or any script file (bat, ps1 or whatever you want) with the incoming argument as a folder path.
# How to set up cookies
1. Open Google Chrome, Microsoft Edge or FireFox.
1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
1. In the opened window, go to ```Application``` - ```Storage``` - ```Cookies```.
@@ -274,11 +361,13 @@ This command can be a batch command or any script file (bat, ps1 or whatever you
**Lines containing a cookie value in curly braces should simply be removed (for example, `{"i_p":0000000000000,"i_l":1}`).**
# How to find Imgur client ID
1. Open browser developer tools
1. Go to ```Network```
1. Go to https://imgur.com/
1. In the list on the left, find a query that contains ```client_id``` and click on it
1. Copy the client id from the ```Request URL```
1. Open settings
1. Paste the copied value into the ```Imgur Clien ID``` field
# How to use the script

14
Users.md

@@ -3,23 +3,18 @@
In the main window, click the ```Add``` button to open the user creation form.
![Add user](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/CreateUserClear.png)
- ```User name``` - you can specify only a username or a link by pattern:
- Twitter:
- https://twitter.com/SomeUserName
- SomeUserName
- Reddit:
- https://reddit.com/user/SomeUserName
- https://reddit.com/r/SomeSubredditName
- u/SomeUserName
- r/SomeSubredditName
- SomeUserName
- Instagram
- https://www.instagram.com/SomeUserName
- SomeUserName
- RedGifs
- https://www.redgifs.com/users/SomeUserName
- SomeUserName
- XVIDEOS
- https://www.xvideos.com/Account-Type/SomeUserName
- Account-Type/SomeUserName
@@ -33,14 +28,13 @@ In the main window, click the ```Add``` button to open the user creation form.
- ```Download images/videos``` - for the created user, only the selected media types will be downloaded.
- ```Ready for download``` - download this user in the operation ```Download all```.
- ```Get user media only``` - this is an option for twitter only. Download all posts posted by this author.
- ```Add by list``` - add users' profiles by list (new line as separator (```Shift``` + ```Enter``` to insert new line)).
- If you add users by the URLs, you must check the ```Auto detect site``` checkbox;
- If you are adding users by usernames, you must select the ```Reddit/Twitter/Instagram``` site.
- ```Add by list``` - add users' profiles by list (new line as separator (```Shift+Enter``` to insert new line)).
- If you add users by the URL, you must check the ```Auto detect site``` checkbox;
- If you are adding users by username, you must select the site in the combo box.
- ```Auto detect site``` - add users by the URLs.
- ```Description``` - some information you want to keep. If the ```Add by list``` checkbox is checked, you must enter a list of users in this textbox (```Shift``` + ```Enter``` to insert new line);
- ```Description``` - some information you want to keep. If the ```Add by list``` checkbox is checked, you must enter a list of users in this textbox (```Shift+Enter``` to insert new line);
- ```Labels``` - labels associated with this user (used to filter users in the main window).
- ```Script``` - If the checkbox is checked, script to be executed after the user download is complete. If the text is empty, the script from the global settings will be executed. [How to use the script](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-use-the-script).
If neither ```Temporary``` nor ```Favorite``` is checked, the user will be displayed in the regular site group.
# Labels