mirror of
https://github.com/AAndyProgram/SCrawler.git
synced 2026-03-15 00:02:17 +00:00
Updated Settings (markdown)
69
Settings.md
69
Settings.md
@@ -1,67 +1,88 @@
|
|||||||
# Settings
|
|
||||||
|
|
||||||
## Program settings
|
## Program settings
|
||||||
|
|
||||||
- Basis
|
- Basis
|
||||||
- ```Data path``` - this is root path where be placed data.
|
- ```Data path``` - this is the root path where the data will be placed.
|
||||||
- ```Large/Small image size``` - size of users' icons.
|
- ```Large/Small image size``` - the size of the user icons.
|
||||||
- ```Collections folder``` - just a collections folder name.
|
- ```Collections folder``` - just the name of the collections folder.
|
||||||
- ```Maximum download tasks of users``` - the number of simultaneously downloading profiles.
|
- ```Maximum download tasks of users``` - the number of simultaneously downloading profiles.
|
||||||
- ```Maximum download tasks of channels``` - the number of simultaneously downloading channels.
|
- ```Maximum download tasks of channels``` - the number of simultaneously downloading channels.
|
||||||
- ```Check new version at start```
|
- ```Check new version at start```
|
||||||
|
- ```Imgur Client ID``` - Imgur client ID to Bypass NSFW protection and download galleries
|
||||||
- Defaults
|
- Defaults
|
||||||
- ```Separate video folder``` - this means video files will be placed in standalone folder in the user folder for keep video files apart of pictures.
|
- ```Separate video folder``` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images.
|
||||||
- ```Temporary``` - this parameter indicates how users will be created by default in the user creation form.
|
- ```Temporary``` - this parameter specifies how users will be created by default in the user creation form.
|
||||||
- ```Download images/videos``` - defaults for creating new users
|
- ```Download images/videos``` - defaults for creating new users
|
||||||
- Channels
|
- Channels
|
||||||
- ```Channels rows/columns``` - how many rows and columns will be shown in the channels form. For example: rows = 2; columns = 5. In this case on one page will be placed 10 images on two rows by 5 columns. **Please don't set up too high value.**
|
- ```Channels rows/columns``` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.**
|
||||||
- ```Download limit for channel user``` - the amount of media will be downloaded if the user added from the channel.
|
- ```Download limit for channel user``` - the amount of media will be downloaded if the user added from the channel.
|
||||||
- ```Copy channel user image``` - the image posted by user in the channel will be copied to the user folder when user will be created.
|
- ```Copy channel user image``` - the image posted by user in the channel will be copied to the user folder when user will be created.
|
||||||
- ```Create temporary users``` - users will be created with 'Temporary' mark when created from channel
|
- ```Create temporary users``` - users will be created marked 'Temporary' when created from channel.
|
||||||
- ```Change file names``` - name files by name (not by original names)
|
- ```Change file names``` - name files by name (not by original names)
|
||||||
- Options:
|
- Options:
|
||||||
- ```Replace file name by date``` - the file name will be replaced with the date of the file was posted
|
- ```Replace file name by date``` - the file name will be replaced with the date of the file was posted
|
||||||
- ```Add date/time to file name``` - the date will be appended to file name
|
- ```Add date/time to file name``` - the date will be appended to the file name
|
||||||
- Parameters:
|
- Parameters:
|
||||||
- ```Date``` - append date to file name
|
- ```Date``` - append date to file name
|
||||||
- ```Time``` - append time to file name
|
- ```Time``` - append time to file name
|
||||||
- Date positions ```Start/End``` - date and/or time will be appended to the end or beginning of the file name
|
- Date positions ```Start/End``` - date and/or time will be appended to the end or beginning of the file name
|
||||||
- Reddit
|
- Reddit
|
||||||
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. It means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
|
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
|
||||||
|
- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
|
||||||
- Twitter
|
- Twitter
|
||||||
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. It means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
|
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
|
||||||
- ```Get user media only``` - default for creating new users
|
- ```Get user media only``` - the default for creating new users
|
||||||
|
- Instagram
|
||||||
|
- ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
|
||||||
|
|
||||||
## Reddit settings
|
## Reddit settings
|
||||||
|
|
||||||
You should set up Reddit path where Reddit users' data will be placed. Default value: ```Data path``` + "Reddit".
|
You need to configure Reddit path where Reddit users' data will be placed. Default value: ```Data path``` + "Reddit".
|
||||||
|
|
||||||
Reddit parser can parse data without cookies, but you can set it up if you want.
|
The Reddit parser can parse data without cookies, but you can add it if you like.
|
||||||
|
|
||||||
## Twitter settings
|
## Twitter settings
|
||||||
|
|
||||||
- ```Path``` - where Twitter users' data will be placed.
|
- ```Path``` - where Twitter users' data will be placed.
|
||||||
- ```Cookies``` add cookies from your browser (press button with pencil image to open cookies editor).
|
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
|
||||||
- ```Token``` - ```x-csrf-token``` must be placed in this field.
|
- ```Token``` - in this field you need to put ```x-csrf-token```.
|
||||||
- ```Authorization``` - Bearer token. Must start with ```Bearer ``` word with space after.
|
- ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space.
|
||||||
|
|
||||||
### How to find tokens
|
### How to find Twitter tokens
|
||||||
|
|
||||||
1. Open Google Chrome or FireFox.
|
1. Open Google Chrome or FireFox.
|
||||||
2. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
|
2. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
|
||||||
3. Open any twitter post you want.
|
3. Open any twitter post you want.
|
||||||
4. In the opened window go to ```Network```.
|
4. In the opened window, go to ```Network```.
|
||||||
5. In the list on the left, find a query that starts with ```TwitterDetails?```... and click on it
|
5. In the list on the left, find a query that starts with ```TwitterDetails?```... and click on it
|
||||||
6. Scroll down to ```Request Headers``` then find the fields below:
|
6. Scroll down to ```Request Headers``` then find the fields below:
|
||||||
- ```authorization: Bearer .......``` - this is the authorization token. You need to copy the text from starting with the word ```Bearer``` to the end of the line and paste it into the ```Authorization``` field on the Twitter settings form.
|
- ```authorization: Bearer .......``` - this is the authorization token. You need to copy the text from starting with the word ```Bearer``` to the end of the line and paste it into the ```Authorization``` field on the Twitter settings form.
|
||||||
- ```x-csrf-token: ...............``` - this is csrf token. You need to copy all the text after the words ```x-csrf-token:``` and paste it into the ```Token``` field on the Twitter settings form
|
- ```x-csrf-token: ...............``` - this is csrf token. You need to copy all the text after the words ```x-csrf-token:``` and paste it into the ```Token``` field on the Twitter settings form
|
||||||
|
|
||||||
|
## Instagram settings
|
||||||
|
|
||||||
|
- ```Path``` - where Instagram users' data will be placed.
|
||||||
|
- ```Cookies``` add cookies from your browser (click the pencil button to open the cookies editor).
|
||||||
|
- ```Hash``` - in this field you need to put the hash of the Instagram session. Just add cookies and click on the curved arrows.
|
||||||
|
|
||||||
|
### Instagram limits
|
||||||
|
|
||||||
|
Instagram API is requests limited. For one request, the program receive only 12 posts. Be careful with ```Ready for download``` option. I recommended parsing one new profile every 5-10 minutes. I'm still thinking about how to make Instagram parsing algorithm smoother.
|
||||||
|
|
||||||
|
I think you can get new posts for several profiles that have already parsed using the ```Download all``` operation without the "Too many requests" error. But if the program catches an error (429), the timeout information will be added to the log.
|
||||||
|
|
||||||
## How to set up cookies
|
## How to set up cookies
|
||||||
|
|
||||||
1. Open Google Chrome.
|
1. Open Google Chrome.
|
||||||
2. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
|
2. Press three-dots-button - More tools - Developer tools (or just press ```Ctrl``` + ```Shift``` + ```I```).
|
||||||
3. In the opened window go to ```Application``` - ```Storage``` - ```Cookies```.
|
3. In the opened window, go to ```Application``` - ```Storage``` - ```Cookies```.
|
||||||
4. Copy all text.
|
4. Copy all text.
|
||||||
5. In the program in the cookies editor press ```From Google Chrome``` button.
|
5. In the program, in the cookie editor, click the button ```From Google Chrome```.
|
||||||
6. Paste copied cookies text in the opened window and press ```OK``` button.
|
6. Paste the copied cookies text into the opened window and click the ```OK``` button.
|
||||||
7. Close cookies editor.
|
7. Close the cookies editor.
|
||||||
|
|
||||||
|
## How to find Imgur client ID
|
||||||
|
|
||||||
|
1. Open browser developer tools
|
||||||
|
2. Go to ```Network```
|
||||||
|
3. In the list on the left, find a query that contains ```client_id``` and click on it
|
||||||
|
4. Copy the client id from the ```Request URL```
|
||||||
Reference in New Issue
Block a user