geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2022.6.3.0

Andy
2022-06-03 20:43:43 +03:00
parent 5a747db867
commit 899bc1aa93
3 changed files with 65 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
Home.md

@@ -60,7 +60,8 @@ This button looks like a bookmark. When you click on this button, a form for dow
- ```Download all site users``` - Download all users marked ```Ready for download``` from specific sites.
- ```Download all FULL``` - Download all users from all sites. **The ```Ready for download``` option will be ignored.**
- ```Download all site users FULL``` - Download all users from specific sites. **The ```Ready for download``` option will be ignored.**
- ```Add a new download group``` -
- ```Add a new download group``` - Create a new download group (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups))
- ```Automation``` - Automatic download settings (read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)
- ```Download video``` - download a separate video (Reddit and Twitter videos supported) or Instagram post (photo and video).
- ```Stop``` - stop all download operations.
@@ -177,30 +178,6 @@ If you want to merge the collection.
Files from the "Reddit_SomeUserName1" and "Twitter_SomeUserName1" folders will be moved to the "First collection" folder.
# Download groups
In many cases, you may need to download some users. You can group these users and make it easier to download them without having to select them every time.
It is very easy to create a new group. Just click the ```Add a new download group``` button (main window - ```Download all``` menu) to create a new one.
- ```Name``` - group name
- ```Temporary``` - users marked as temporary (indeterminate state to not use this option) will be downloaded
- ```Favorite``` - users marked as favorite (indeterminate state to not use this option) will be downloaded
- ```Ready for download``` - users marked as ```Ready for download``` will be downloaded
- ```Ignore ready for download``` - this option tells the program to ignore the ```Ready for download``` user option and download the user anyway
- ```Labels``` - You can select labels. Only users who have one or more of these labels will be downloaded.
**Only those users who match all of these parameters (logical operator ```AND```) will be downloaded.**
![Group creator](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/GroupCreating.png)
For each group, SCrawler creates a new menu, which is placed in the ```Download all``` menu of the main window. If a group has a number on the left (1-9), that group can be downloaded using ```Ctrl```+```Number```. Each group also has several options:
- ```Edit``` - edit group
- ```Delete``` - delete group
- ```Download``` - download with the options you have set
- ```Download FULL``` - Download with the options you have set. ```Ready for download``` option will be ignored.
![Groups context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
# Black list
The program has the ```Black list```, which included users you have added to. It is only a blacklist of the program and no impact on the site!

4
Plugins.md

@@ -245,11 +245,13 @@ Called from the settings root when the settings ```EndUpdate``` function is call
### Available
```Function Available(ByVal What As Download) As Boolean```
```Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean```
If your plugin needs to be checked for availability (you can see an example in the [```API.Reddit.SiteSettings```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Reddit/SiteSettings.vb)), place the check code here. Otherwise, this function **MUST** return ```true```.
If this function returns ```false```, users of your plugin site will not be downloaded.
```Silent``` stands for show notification or not.
### ReadyToDownload
```Function ReadyToDownload(ByVal What As Download) As Boolean```

60
Settings.md

@@ -50,6 +50,10 @@
![Default settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsGlobalDownloading.png)
## Automation settings
**Read more [here](#automation)**
## Channels
- ```Channels rows/columns``` - how many rows and columns will be displayed in the channels form. For example: rows = 2; columns = 5. In this case, 10 images will be placed on one page in two rows of 5 columns. **Please don't set too high value.**
- ```Download limit for channel user``` - the amount of media will be downloaded if the user added from the channel.
@@ -70,6 +74,7 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- ```Get user media only``` - the default for creating new users
- ```Saved posts path``` - if you wish, you can specify a special path for saved posts. Leave it blank to use the default path.
- ```Saved posts user``` - your personal Reddit username to download your saved posts (this feature requires cookies)
- ```Use M3U8``` - If you are using an x86 program, you will not be able to use the M3U8 download. In other cases, this setting controls how to download videos hosted on Reddit. There are two ways to download: the mp4 file as is, but without an audio track, and through an m3u8 playlist that contains an audio track.
![Reddit settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteReddit.png)
@@ -121,6 +126,7 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- ```Post limit timer``` - this is the time value (in milliseconds) the program will wait before processing the next request after 195 requests (**it is highly recommended not to change this default value**)
- ```Get stories```
- ```Get tagged photos```
- ```Tagged notify limit``` - Limit of new tagged posts when you receive a notification (read more [here](#instagram-tagged-posts-limit))
![Instagram settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsSiteInstagram.png)
@@ -145,11 +151,65 @@ The Reddit parser can parse data without cookies, but you can add it if you like
### Instagram limits
Instagram API is requests limited. For one request, the program receive only 50 posts. Before catching error 429, the program can process 200 requests. I reduced this to 195 requests and set a timer to wait for the next request after. This was added to bypass error 429 and prevent account ban.
### Instagram tagged posts limit
When the number of tagged posts exceeds the tagged posts limit, you will be asked what to do about it. This message box protects you from account ban and long wait times.
**ATTENTION. Change the default parameters at your own risk. If you do this, I don't guarantee that your account will not be banned.**
- ```Continue``` - Continue downloading
- ```Continue unnotified``` - Continue downloading and cancel further notifications in the current downloading session
- ```Limit``` - Enter the limit of posts you want to download
- ```Confirm``` - Confirm the number you entered
- ```Try again``` - You will be asked again about the limit
- ```Other options``` - The main message with options will be displayed again
- ```Cancel``` - Cancel tagged posts download operation
- ```Disable and cancel``` - Disable downloading tagged data and cancel downloading tagged data
- ```Cancel``` - Cancel tagged posts download operation
## RedGifs
```Temporary```, ```Download images/videos``` - personal site settings by default for creating new users. Means the same as the parameters of the same name on the ```Defaults``` tab. If the checkbox is in an intermediate state, then the same name setting of the ```Defaults``` tab will be used instead. Otherwise, this setting will be applied.
# Download groups
In many cases, you may need to download some users. You can group these users and make it easier to download them without having to select them every time.
It is very easy to create a new group. Just click the ```Add a new download group``` button (main window - ```Download all``` menu) to create a new one.
- ```Name``` - group name
- ```Temporary``` - users marked as temporary (indeterminate state to not use this option) will be downloaded
- ```Favorite``` - users marked as favorite (indeterminate state to not use this option) will be downloaded
- ```Ready for download``` - users marked as ```Ready for download``` will be downloaded
- ```Ignore ready for download``` - this option tells the program to ignore the ```Ready for download``` user option and download the user anyway
- ```Labels``` - You can select labels. Only users who have one or more of these labels will be downloaded.
**Only those users who match all of these parameters (logical operator ```AND```) will be downloaded.**
![Group creator](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/GroupCreating.png)
For each group, SCrawler creates a new menu, which is placed in the ```Download all``` menu of the main window. If a group has a number on the left (1-9), that group can be downloaded using ```Ctrl```+```Number```. Each group also has several options:
- ```Edit``` - edit group
- ```Delete``` - delete group
- ```Download``` - download with the options you have set
- ```Download FULL``` - Download with the options you have set. ```Ready for download``` option will be ignored.
![Groups context](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/MainWindowGroups.png)
# Automation
You can set up automatic downloads. You can find these settings in ```Settings-Automation``` or by clicking the ```Automation``` button in the ```Download all``` drop-down list in the main window.
- ```Disable``` - Disable automatic downloads
- ```All``` - Download all existing user profiles (```Ready for download``` is not respected)
- ```Default``` - Download all existing user profiles marked ```Ready for download``` (equivalent to ```Download all```)
- ```Specified``` - Specify custom download options (read more [here](#download-groups))
- ```Groups``` - Automatic downloading of one or more existing groups (read more here)
- ```Show notifications``` - Show notification when there is new downloaded data
- ```Timer``` - Download timer (in minutes)
![Automation settings](https://github.com/AAndyProgram/SCrawler/blob/main/ProgramScreenshots/SettingsAutoDownloader.png)
# Folder command
You can specify a command to open a folder in a special program. Pattern: ```Command "{0}"```. ```"{0}"``` is the path argument. I like to use total commander and I want to open folders in this program. Example: ```D:\TotalCMD\TOTALCMD64.EXE /o "{0}"```.