diff --git a/Settings.md b/Settings.md index 3f82738..a5ce5ab 100644 --- a/Settings.md +++ b/Settings.md @@ -1,4 +1,4 @@ -## Program settings +# Program settings - Basis - ```Data path``` - this is the root path where the data will be placed. @@ -19,46 +19,52 @@ - ```Exit confirm``` - ask for confirmation before closing the program - ```Close to tray``` - close program to system tray - ```Show notifications``` - show notifications when download is complete + - ```Fast profiles loading``` - fast loading profiles in the main window. **Be careful with this setting. Fast loading leads to the highest CPU usage.** + - ```Folder cmd``` - the [command](#folder-command) to open a folder + + ![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsBasis.png) - Defaults - ```Separate video folder``` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images. - ```Temporary``` - this parameter specifies how users will be created by default in the user creation form. - ```Download images/videos``` - defaults for creating new users + - ```Update user description every time``` - this means that the user description (if implemented) will be checked when the user is parsed. If the current user description does not contain a new description, then a new one will be added via a new line. + + ![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingDefaults.png) + - Channels - ```Channels rows/columns``` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.** - ```Download limit for channel user``` - the amount of media will be downloaded if the user added from the channel. - ```Copy channel user image``` - the image posted by user in the channel will be copied to the user folder when user will be created. - ```Create temporary users``` - users will be created marked 'Temporary' when created from channel. -- Reddit - - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. - - ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies) - - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. -- Twitter - - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. - - ```Get user media only``` - the default for creating new users -- Instagram - - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. - - ```Request timer``` - this is the time value (in milliseconds) the program will wait before processing the next ```Request time counter``` request (**it is highly recommended not to change this default value**) - - ```Request time counter``` - how many requests will be sent to Instagram before the program waits ```Request timer``` milliseconds (**it is highly recommended not to change this default value**) - - ```Post limit timer``` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**) - - ```Saved posts user``` - your personal Instagram username to download your saved posts (this feature requires cookies and **InstaHash 2**) - - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. -- RedGifs - - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. -## Reddit settings + ![Channels settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsChannels.png) + +# Site settings + +## Reddit You need to configure Reddit path where Reddit users' data will be placed. Default value: ```Data path``` + "Reddit". The Reddit parser can parse data without cookies, but you can add it if you like. -## Twitter settings + - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. + - ```Get user media only``` - the default for creating new users + - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. + - ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies) -- ```Path``` - where Twitter users' data will be placed. -- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor). -- ```Token``` - in this field you need to put ```x-csrf-token```. -- ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space. + ![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsReddit.png) +## Twitter + - ```Path``` - where Twitter users' data will be placed. + - ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor). + - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. + - ```Get user media only``` - the default for creating new users + - ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space. + - ```Token``` - in this field you need to put ```x-csrf-token```. + + ![Twitter settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsTwitter.png) + ### How to find Twitter tokens 1. Open Google Chrome, Edge or FireFox. @@ -71,21 +77,44 @@ The Reddit parser can parse data without cookies, but you can add it if you like - ```authorization: Bearer .......``` - this is the authorization token. You need to copy the text from starting with the word ```Bearer``` to the end of the line and paste it into the ```Authorization``` field on the Twitter settings form. - ```x-csrf-token: ...............``` - this is csrf token. You need to copy all the text after the words ```x-csrf-token:``` and paste it into the ```Token``` field on the Twitter settings form -## Instagram settings + +## Instagram -- ```Path``` - where Instagram users' data will be placed. -- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor). -- ```Hash``` - in this field you need to put the hash of the Instagram session. Just add cookies and click on the curved arrows. -- ```Hash 2``` - in this field you need to put the [hash of the Instagram session for saved posts](#how-to-find-instagram-hash-2). + - ```Path``` - where Instagram users' data will be placed. + - ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor). + - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. + - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. + - Authorization + - ```Hash``` - in this field you need to put the hash of the Instagram session. Just add cookies and click on the curved arrows. + - ```Hash 2``` - in this field you need to put the [hash of the Instagram session for saved posts](#how-to-find-instagram-hash-2). + - ```x-ig-app-id```, ```ix-ig-www-claim``` - read [here](#how-to-find-instagram-stories-authorization-headers) how to find them + - ```Saved posts user``` - your personal Instagram username to download your saved posts (this feature requires cookies and **InstaHash 2**) + - Other parameters + - ```Request timer``` - this is the time value (in milliseconds) the program will wait before processing the next ```Request time counter``` request (**it is highly recommended not to change this default value**) + - ```Request time counter``` - how many requests will be sent to Instagram before the program waits ```Request timer``` milliseconds (**it is highly recommended not to change this default value**) + - ```Post limit timer``` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**) + - ```Get stories``` + - ```Get tagged photos``` + + ![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsInstagram.png) ### How to find Instagram Hash 2 1. Open browser developer tools -2. Go to ```Network``` -3. If this tab contain any data, clear it. -4. Open your saved Instagram posts. -5. Click ```Ctrl``` + ```F``` and find the text ```query_hash=```. The value (of letters and numbers) after the equal sign is the required hash. -6. Copy this and paste it into the ```Hash 2``` field on the Instagram settings form. +1. Go to ```Network``` +1. If this tab contain any data, clear it. +1. Open your saved Instagram posts. +1. Click ```Ctrl``` + ```F``` and find the text ```query_hash=```. The value (of letters and numbers) after the equal sign is the required hash. +1. Copy this and paste it into the ```Hash 2``` field on the Instagram settings form. + +### How to find Instagram stories authorization headers + +1. Open browser developer tools +1. Go to ```Network``` +1. If this tab contain any data, clear it. +1. Open your saved Instagram tagged photos. +1. Click ```Ctrl``` + ```F``` and find the text ```x-ig-app-id``` and ```x-ig-www-claim```. +1. Copy the values and paste them into the corresponding fields on the Instagram settings form. ### Instagram limits @@ -94,20 +123,31 @@ Instagram API is requests limited. For one request, the program receive only 50 ~~Be careful with ```Ready for download``` option. I recommended parsing one new profile every 5-10 minutes. I'm still thinking about how to make Instagram parsing algorithm smoother.~~ ~~I think you can get new posts for several profiles that have already parsed using the ```Download all``` operation without the "Too many requests" error. But if the program catches an error (429), the timeout information will be added to the log.~~ + +## RedGifs + - ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. -## How to set up cookies +# Folder command + +You can specify a command to open a folder in a special program. Pattern: ```Command "{0}"```. ```"{0}"``` is the path argument. I like to use total commander and I want to open folders in this program. Example: ```D:\TotalCMD\TOTALCMD64.EXE /o "{0}"```. + +```"{0}"``` - required argument! + +This command can be a batch command or any script file (bat, ps1 or whatever you want) with the incoming argument as a folder path. + +# How to set up cookies 1. Open Google Chrome, Microsoft Edge or FireFox. -2. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```). -3. In the opened window, go to ```Application``` - ```Storage``` - ```Cookies```. -4. Copy all text **with mouse**. -5. In the program, in the cookie editor, click the button ```From Google Chrome```. -6. Paste the copied cookies text into the opened window and click the ```OK``` button. -7. Close the cookies editor. +1. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```). +1. In the opened window, go to ```Application``` - ```Storage``` - ```Cookies```. +1. Copy all text **with mouse**. +1. In the program, in the cookie editor, click the button ```From Google Chrome```. +1. Paste the copied cookies text into the opened window and click the ```OK``` button. +1. Close the cookies editor. -## How to find Imgur client ID +# How to find Imgur client ID 1. Open browser developer tools -2. Go to ```Network``` -3. In the list on the left, find a query that contains ```client_id``` and click on it -4. Copy the client id from the ```Request URL``` \ No newline at end of file +1. Go to ```Network``` +1. In the list on the left, find a query that contains ```client_id``` and click on it +1. Copy the client id from the ```Request URL``` \ No newline at end of file