geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-15 00:02:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

3.0.0.0

Andy
2022-03-17 21:24:20 +03:00
parent 66b82832b7
commit d039fb9d33
1 changed files with 84 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

128
Settings.md

@@ -1,4 +1,4 @@
## Program settings
# Program settings
- Basis
- ```Data path``` - this is the root path where the data will be placed.
@@ -19,46 +19,52 @@
- ```Exit confirm``` - ask for confirmation before closing the program
- ```Close to tray``` - close program to system tray
- ```Show notifications``` - show notifications when download is complete
- ```Fast profiles loading``` - fast loading profiles in the main window. **Be careful with this setting. Fast loading leads to the highest CPU usage.**
- ```Folder cmd``` - the [command](#folder-command) to open a folder
![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsBasis.png)
- Defaults
- ```Separate video folder``` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images.
- ```Temporary``` - this parameter specifies how users will be created by default in the user creation form.
- ```Download images/videos``` - defaults for creating new users
- ```Update user description every time``` - this means that the user description (if implemented) will be checked when the user is parsed. If the current user description does not contain a new description, then a new one will be added via a new line.
![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingDefaults.png)
- Channels
- ```Channels rows/columns``` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.**
- ```Download limit for channel user``` - the amount of media will be downloaded if the user added from the channel.
- ```Copy channel user image``` - the image posted by user in the channel will be copied to the user folder when user will be created.
- ```Create temporary users``` - users will be created marked 'Temporary' when created from channel.
- Reddit
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
- Twitter
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- ```Get user media only``` - the default for creating new users
- Instagram
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- ```Request timer``` - this is the time value (in milliseconds) the program will wait before processing the next ```Request time counter``` request (**it is highly recommended not to change this default value**)
- ```Request time counter``` - how many requests will be sent to Instagram before the program waits ```Request timer``` milliseconds (**it is highly recommended not to change this default value**)
- ```Post limit timer``` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
- ```Saved posts user``` - your personal Instagram username to download your saved posts (this feature requires cookies and **InstaHash 2**)
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
- RedGifs
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
## Reddit settings
![Channels settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsChannels.png)
# Site settings
## Reddit
You need to configure Reddit path where Reddit users' data will be placed. Default value: ```Data path``` + "Reddit".
The Reddit parser can parse data without cookies, but you can add it if you like.
## Twitter settings
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- ```Get user media only``` - the default for creating new users
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
- ```Path``` - where Twitter users' data will be placed.
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
- ```Token``` - in this field you need to put ```x-csrf-token```.
- ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space.
![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsReddit.png)
## Twitter
- ```Path``` - where Twitter users' data will be placed.
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- ```Get user media only``` - the default for creating new users
- ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space.
- ```Token``` - in this field you need to put ```x-csrf-token```.
![Twitter settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsTwitter.png)
### How to find Twitter tokens
1. Open Google Chrome, Edge or FireFox.
@@ -71,21 +77,44 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- ```authorization: Bearer .......``` - this is the authorization token. You need to copy the text from starting with the word ```Bearer``` to the end of the line and paste it into the ```Authorization``` field on the Twitter settings form.
- ```x-csrf-token: ...............``` - this is csrf token. You need to copy all the text after the words ```x-csrf-token:``` and paste it into the ```Token``` field on the Twitter settings form
## Instagram settings
## Instagram
- ```Path``` - where Instagram users' data will be placed.
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
- ```Hash``` - in this field you need to put the hash of the Instagram session. Just add cookies and click on the curved arrows.
- ```Hash 2``` - in this field you need to put the [hash of the Instagram session for saved posts](#how-to-find-instagram-hash-2).
- ```Path``` - where Instagram users' data will be placed.
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
- Authorization
- ```Hash``` - in this field you need to put the hash of the Instagram session. Just add cookies and click on the curved arrows.
- ```Hash 2``` - in this field you need to put the [hash of the Instagram session for saved posts](#how-to-find-instagram-hash-2).
- ```x-ig-app-id```, ```ix-ig-www-claim``` - read [here](#how-to-find-instagram-stories-authorization-headers) how to find them
- ```Saved posts user``` - your personal Instagram username to download your saved posts (this feature requires cookies and **InstaHash 2**)
- Other parameters
- ```Request timer``` - this is the time value (in milliseconds) the program will wait before processing the next ```Request time counter``` request (**it is highly recommended not to change this default value**)
- ```Request time counter``` - how many requests will be sent to Instagram before the program waits ```Request timer``` milliseconds (**it is highly recommended not to change this default value**)
- ```Post limit timer``` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
- ```Get stories```
- ```Get tagged photos```
![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsInstagram.png)
### How to find Instagram Hash 2
1. Open browser developer tools
2. Go to ```Network```
3. If this tab contain any data, clear it.
4. Open your saved Instagram posts.
5. Click ```Ctrl``` + ```F``` and find the text ```query_hash=```. The value (of letters and numbers) after the equal sign is the required hash.
6. Copy this and paste it into the ```Hash 2``` field on the Instagram settings form.
1. Go to ```Network```
1. If this tab contain any data, clear it.
1. Open your saved Instagram posts.
1. Click ```Ctrl``` + ```F``` and find the text ```query_hash=```. The value (of letters and numbers) after the equal sign is the required hash.
1. Copy this and paste it into the ```Hash 2``` field on the Instagram settings form.
### How to find Instagram stories authorization headers
1. Open browser developer tools
1. Go to ```Network```
1. If this tab contain any data, clear it.
1. Open your saved Instagram tagged photos.
1. Click ```Ctrl``` + ```F``` and find the text ```x-ig-app-id``` and ```x-ig-www-claim```.
1. Copy the values and paste them into the corresponding fields on the Instagram settings form.
### Instagram limits
@@ -94,20 +123,31 @@ Instagram API is requests limited. For one request, the program receive only 50
~~Be careful with ```Ready for download``` option. I recommended parsing one new profile every 5-10 minutes. I'm still thinking about how to make Instagram parsing algorithm smoother.~~
~~I think you can get new posts for several profiles that have already parsed using the ```Download all``` operation without the "Too many requests" error. But if the program catches an error (429), the timeout information will be added to the log.~~
## RedGifs
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
## How to set up cookies
# Folder command
You can specify a command to open a folder in a special program. Pattern: ```Command "{0}"```. ```"{0}"``` is the path argument. I like to use total commander and I want to open folders in this program. Example: ```D:\TotalCMD\TOTALCMD64.EXE /o "{0}"```.
```"{0}"``` - required argument!
This command can be a batch command or any script file (bat, ps1 or whatever you want) with the incoming argument as a folder path.
# How to set up cookies
1. Open Google Chrome, Microsoft Edge or FireFox.
2. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
3. In the opened window, go to ```Application``` - ```Storage``` - ```Cookies```.
4. Copy all text **with mouse**.
5. In the program, in the cookie editor, click the button ```From Google Chrome```.
6. Paste the copied cookies text into the opened window and click the ```OK``` button.
7. Close the cookies editor.
1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
1. In the opened window, go to ```Application``` - ```Storage``` - ```Cookies```.
1. Copy all text **with mouse**.
1. In the program, in the cookie editor, click the button ```From Google Chrome```.
1. Paste the copied cookies text into the opened window and click the ```OK``` button.
1. Close the cookies editor.
## How to find Imgur client ID
# How to find Imgur client ID
1. Open browser developer tools
2. Go to ```Network```
3. In the list on the left, find a query that contains ```client_id``` and click on it
4. Copy the client id from the ```Request URL```
1. Go to ```Network```
1. In the list on the left, find a query that contains ```client_id``` and click on it
1. Copy the client id from the ```Request URL```