geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

3.0.0.9

Andy
2022-04-24 20:36:17 +03:00
parent 687b583904
commit c985d04ac8
2 changed files with 55 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
Home.md

@@ -79,9 +79,11 @@ This button looks like a bookmark. When you click on this button, a form for dow
- ```Favorite``` - only show users marked as favorite. - ```Favorite``` - only show users marked as favorite.
- ```Deleted``` - only show users whose profiles have been deleted - ```Deleted``` - only show users whose profiles have been deleted
- ```Suspended``` - only show users whose profiles have been suspended - ```Suspended``` - only show users whose profiles have been suspended
- ```Labels``` - show users labeled with one or more selected label (click ```Select labels``` to select labels). - ```Labels``` - show users labeled with one or more selected label
- ```No Labels``` - only show users without a label. - ```No Labels``` - only show users without a label
- ```Select labels``` - select one or more labels to display. - ```Excluded labels``` - exclude from displaying users with the selected labels (this is suitable for situations where you want to hide a user but don't want to delete)
- ```Ignore excluded labels``` - Ignore excluding labels. This allows you to show users that should be excluded from display without resetting the current selection of excluded labels.
- ```Show groups instead of labels``` - show sites groups instead of labels (when filtering users by labels)
- ```Limit dates``` - set lowest date to filter out users who have not been downloaded since that date - ```Limit dates``` - set lowest date to filter out users who have not been downloaded since that date
![Main window context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindow2.png) ![Main window context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindow2.png)

83
Settings.md

@@ -1,14 +1,39 @@
# Program settings # Program settings
- Basis ## Basis
- ```Data path``` - this is the root path where the data will be placed. - ```Data path``` - this is the root path where the data will be placed.
- ```Large/Small image size``` - the size of the user icons. - ```Large/Small image size``` - the size of the user icons.
- ```Collections folder``` - just the name of the collections folder. - ```Collections folder``` - just the name of the collections folder.
- ```Maximum download tasks of users``` - the number of simultaneously downloading profiles. - ```Maximum download tasks of users``` - the number of simultaneously downloading profiles.
- ```Maximum download tasks of channels``` - the number of simultaneously downloading channels. - ```Maximum download tasks of channels``` - the number of simultaneously downloading channels.
- ```Check new version at start``` - ```Check new version at start```
- ```Imgur Client ID``` - Imgur client ID to Bypass NSFW protection and download galleries - ```Imgur Client ID``` - Imgur client ID to Bypass NSFW protection and download galleries
- ```Change file names``` - name files by name (not by original names) - ```Show groups``` - Show user site groups
- ```Use user grouping``` - Group users by groups and/or labels. If disabled, users will not be grouped by site or label.
![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBasis.png)
## Behavior
- ```Exit confirm``` - ask for confirmation before closing the program
- ```Close to tray``` - close program to system tray
- ```Show notifications``` - show notifications when download is complete
- ```Fast profiles loading``` - fast loading profiles in the main window. **Be careful with this setting. Fast loading leads to the highest CPU usage.**
- ```Delete data to recycle bin``` - delete data to recycle bin or permanent
- ```Folder cmd``` - the [command](#folder-command) to open a folder
![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalBehavior.png)
## Defaults
- ```Separate video folder``` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images.
- ```Temporary``` - this parameter specifies how users will be created by default in the user creation form.
- ```Download images/videos``` - defaults for creating new users
![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDefaults.png)
## Downloading
- ```Update user description every time``` - this means that the user description (if implemented) will be checked when the user is parsed. If the current user description does not contain a new description, then a new one will be added via a new line.
- ```Change file names``` - name files by name (not by original names)
- Options: - Options:
- ```Replace file name by date``` - the file name will be replaced with the date of the file was posted - ```Replace file name by date``` - the file name will be replaced with the date of the file was posted
- ```Add date/time to file name``` - the date will be appended to the file name - ```Add date/time to file name``` - the date will be appended to the file name
@@ -16,31 +41,17 @@
- ```Date``` - append date to file name - ```Date``` - append date to file name
- ```Time``` - append time to file name - ```Time``` - append time to file name
- Date positions ```Start/End``` - date and/or time will be appended to the end or beginning of the file name - Date positions ```Start/End``` - date and/or time will be appended to the end or beginning of the file name
- ```Exit confirm``` - ask for confirmation before closing the program - ```Script``` - [script](#how-to-use-the-script) to be executed after the user download is complete. If the checkbox is checked, new users will be created with the ```Use script``` option.
- ```Close to tray``` - close program to system tray
- ```Show notifications``` - show notifications when download is complete
- ```Fast profiles loading``` - fast loading profiles in the main window. **Be careful with this setting. Fast loading leads to the highest CPU usage.**
- ```Folder cmd``` - the [command](#folder-command) to open a folder
- ```Delete data to recycle bin``` - delete data to recycle bin or permanent
- ```Script``` - [script](#how-to-use-the-script) to be executed after the user download is complete. If the checkbox is checked, new users will be created with the ```Use script``` option.
![Basis settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsBasis.png) ![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDownloading.png)
- Defaults ## Channels
- ```Separate video folder``` - this means that video files will be placed in a separate folder in the user's folder to store video files separately from images. - ```Channels rows/columns``` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.**
- ```Temporary``` - this parameter specifies how users will be created by default in the user creation form. - ```Download limit for channel user``` - the amount of media will be downloaded if the user added from the channel.
- ```Download images/videos``` - defaults for creating new users - ```Copy channel user image``` - the image posted by user in the channel will be copied to the user folder when user will be created.
- ```Update user description every time``` - this means that the user description (if implemented) will be checked when the user is parsed. If the current user description does not contain a new description, then a new one will be added via a new line. - ```Create temporary users``` - users will be created marked 'Temporary' when created from channel.
![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingDefaults.png) ![Channels settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalChannels.png)
- Channels
- ```Channels rows/columns``` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.**
- ```Download limit for channel user``` - the amount of media will be downloaded if the user added from the channel.
- ```Copy channel user image``` - the image posted by user in the channel will be copied to the user folder when user will be created.
- ```Create temporary users``` - users will be created marked 'Temporary' when created from channel.
![Channels settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsChannels.png)
# Site settings # Site settings
@@ -55,7 +66,7 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path. - ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies) - ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsReddit.png) ![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteReddit.png)
## Reddit user settings ## Reddit user settings
@@ -72,7 +83,7 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space. - ```Authorization``` - Bearer token. Must start with ```Bearer ``` word followed by a space.
- ```Token``` - in this field you need to put ```x-csrf-token```. - ```Token``` - in this field you need to put ```x-csrf-token```.
![Twitter settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsTwitter.png) ![Twitter settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteTwitter.png)
### How to find Twitter tokens ### How to find Twitter tokens
@@ -105,7 +116,7 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- ```Get stories``` - ```Get stories```
- ```Get tagged photos``` - ```Get tagged photos```
![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsInstagram.png) ![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteInstagram.png)
### How to find Instagram Hash 2 ### How to find Instagram Hash 2
@@ -129,10 +140,6 @@ The Reddit parser can parse data without cookies, but you can add it if you like
Instagram API is requests limited. For one request, the program receive only 50 posts. Before catching error 429, the program can process 200 requests. I reduced this to 195 requests and set a timer to wait for the next request after. Instagram API is requests limited. For one request, the program receive only 50 posts. Before catching error 429, the program can process 200 requests. I reduced this to 195 requests and set a timer to wait for the next request after.
~~Be careful with ```Ready for download``` option. I recommended parsing one new profile every 5-10 minutes. I'm still thinking about how to make Instagram parsing algorithm smoother.~~
~~I think you can get new posts for several profiles that have already parsed using the ```Download all``` operation without the "Too many requests" error. But if the program catches an error (429), the timeout information will be added to the log.~~
## RedGifs ## RedGifs
```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied. ```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.