geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-15 00:02:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

3.0.0.0

Andy
2022-03-17 21:25:53 +03:00
parent c8766ce10a
commit e3207efec5
1 changed files with 540 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

540
Plugins.md Normal file

@@ -0,0 +1,540 @@
You can develop your plugin for any site you want. I created a plugin environment with full SCrawler integration. You can display any class properties in SCrawler's internal edit forms.
The class that contains all the available objects is the Instagram settings:
- [```SiteSettings```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/SiteSettings.vb)
- [```UserData```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/UserData.vb)
- [```SavedPosts```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Instagram/ProfileSaved.vb)
- [```SiteSettingsBase```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Base/SiteSettingsBase.vb)
You can look at them as an example.
The ```SCrawler.PluginProvider.dll``` is fully compatible with the most popular language - CSharp.
Net.Framework version is 4.6.1.
# Plugins environment
The plugin concept is based on two main interfaces. The first interface - [```ISiteSettings```](#isitesettings) - is the major interface of the plugin. This class **must** provide all instances, validators, and other objects required by SCrawler. The second interface - [```IPluginContentProvider```](#iplugincontentprovider) - is an instance of UserData.
# How to make a plugin
1.
1. Create a new Net.Framework library project.
1. Add a reference to ```SCrawler.PluginProvider.dll```
1.
1. Create a new settings class (e.g. ```SiteSettings```).
1. Implement the ```ISiteSettings``` interface.
1. Add the required ```Manifest``` [class attribute](#manifest).
1. Add optional attributes if you need: [```SeparatedTasks```](#separatedtasks), [```SavedPosts```](#savedposts), [```SpecialForm```](#specialform)
1.
- Set values for [Site](#site), [Icon](#icon), [Image](#image).
- ```Site``` is a required property (cannot be null)
- If your plugin doesn't have an ```Icon``` or ```Image```, these properties must return ```Nothing``` (```null``` in C#).
- **Attention! Don't add any attributes to these properties!**
1. Define your own properties if you need to. To allow users to change the property values of your class and interact with SCrawler, this property must be an AutoProperty and must be declared as ```PropertyValue``` [object](#propertyvalue).
1. Write the code for the interface functions. **If a function has a return type, then that function MUST return a value!**
1.
- Develop a user instance of the ```IPluginContentProvider``` interface
- This instance must be returned by the [```GetInstance```](#getinstance) function
1. If you plugin has special options, develop an options exchange class and options editor form.
1. It is done. Your plugin is ready to work. :blush:
# How it works
1. When launched, SCrawler loads plugins from the ```Plugins``` folder.
1. A new [```SettingsHost```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/PluginsEnvironment/Hosts/SettingsHost.vb) is created for each plugin.
1. Get class attributes
1. Invoke [```BeginInit```](#begininit) function
1. Sending XML data to your plugin class using the [```Load```](#load) function.
1. Get class members
1. Create hosts for [```PropertyValue```](#propertyvalue)
1. Adding [property updaters](#propertyupdater) to the property host
1. Adding [property providers](#provider) to the property host
1. Adding [property data checkers](#propertiesdatachecker) to the property host
1. Invoke [```EndInit```](#begininit) function
1. Get property values stored in XML (and create new ones if they don't exist) and replace default values with XML values.
1. Creating a list of users using the [```GetInstance```](#getinstance) function
- For each user functions [```XmlFieldsSet```](#xmlfieldsset) and [```XmlFieldsGet```](#xmlfieldsget) will be called
1. When downloading
1. If [```Available(Main)```](#available) = ```false```, users of your plugin will not be downloaded.
1. Invoke [```DownloadStarted(Main)```](#downloadstarted)
1. Per downloading user
1. If [```ReadyToDownload(Main)```](#readytodownload) = ```true```
1. Invoke [```BeforeStartDownload(User, Main)```](#beforestartdownload)
1. Downloading data
1. Sending information to your instance:
- ```Settings``` = your settings instance
- ```Thrower```
- ```LogProvider```
- ```Name```
- ```ID```
- ```ParseUserMediaOnly```
- ```UserDescription```
- ```UserExists```
- ```UserSuspended```
- ```IsSavedPosts```
- ```SeparateVideoFolder```
- ```DataPath```
- ```PostsNumberLimit```
- ```ExistingContentList``` = list of existing content stored in user files
- ```TempPostsList``` = list of post IDs stored in user files
1. Invoke [```GetMedia```](#getmedia)
1. Get the ```TempPostsList``` list back
1. Get other parameters back
- ```Name```
- ```ID```
- ```UserDescription```
- ```UserExists```
- ```UserSuspended```
1. If the instance does not have [```UseInternalDownloader```](#useinternaldownloader) attribute, invoke [```Download```](#download) function
1. Invoke [```AfterDownload(User, Main)```](#afterdownload)
1. Invoke [```DownloadDone(Main)```](#downloaddone)
1. When downloading saved posts
1. If [```ReadyToDownload(SavedPosts)```](#readytodownload) = ```true```
1. If [```Available(SavedPosts)```](#available) = ```true```
1. Invoke [```DownloadStarted(SavedPosts)```](#downloadstarted)
1. Get an instance with [```GetInstance(SavedPosts)```](#getinstance). The Instance must contain the UserName (```Name```)
1. Set ```IsSavedPosts``` = ```true```
1. Invoke [```BeforeStartDownload(User, SavedPosts)```](#beforestartdownload)
1. The same steps as in: ```When downloading``` - ```Per downloading user``` - ```Downloading data```
1. Invoke [```AfterDownload(User, SavedPosts)```](#afterdownload)
1. Invoke [```DownloadDone(SavedPosts)```](#downloaddone)
1. When user creating / updating
1. If your class has special options ([```SpecialForm(False)```](#specialform)), the ```Options``` button will be enabled
1. [```UserOptions(OptionsArg, False)```](#useroptions) will be invoked. If ```OptionsArg``` is ```Nothing``` (```null``` in C#), you **MUST** create a new exchange object.
1. By clicking on the ```Options``` button, [```UserOptions(OptionsArg, True)```](#useroptions) will be invoked. If ```OptionsArg``` is ```Nothing``` (```null``` in C#), you **MUST** create a new exchange object. Then open the form to change the settings.
1. When updating a user, [```ExchangeOptionsGet```](#exchangeoptionsget) will be invoked. You **MUST** return a new exchange object with user-saved options. At the end of editing the user, [```ExchangeOptionsSet```](#exchangeoptionsset) will be invoked with your exchange object. You **MUST** apply the options of your exchange object to the user.
# Interfaces
## ISiteSettings
**Don't duplicate the ```Temporary```, ```Favorite```, ```Path``` and ```SavedPostsPath``` properties in the settings class. These properties are basic host data.**
C# explanation
``` CSharp
interface ISiteSettings
{
Icon Icon {get;};
Image Image {get;};
string Site {get;};
string GetUserUrl(string UserName,bool Channel);
ExchangeOptions IsMyUser(string UserURL);
ExchangeOptions IsMyImageVideo(string URL);
IEnumerable<IPluginUserMedia> GetSpecialData(string URL);
IPluginContentProvider GetInstance(Download What);
void Load(IEnumerable<KeyValuePair<string, string>> XMLValues);
IEnumerable<KeyValuePair<string, string>> XMLFields();
void BeginInit();
void EndInit();
bool Available();
bool ReadyToDownload();
void DownloadStarted(Download What);
void BeforeStartDownload(object User, Download What);
void AfterDownload(object User, Download What);
void DownloadDone(Download What);
void OpenSettingsForm();
void UserOptions(ref object Options, bool OpenForm);
}
```
### Site
Site name. This ReadOnly property must only provide the name of the site. For example: for instagram.com you should set the site name ```Instagram```. The first letter must be uppercase.
### Icon
Site icon for forms and other objects (can be null).
### Image
Site image for controls (such as buttons) and other objects.
### GetUserUrl
```Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String```
This function should return the user's URL based on the ```UserName``` argument. This function is called when the user clicks the ```Open site``` button in SCrawler.
The ```Channel``` argument specifies that the user is the a channel.
### IsMyUser
```Function IsMyUser(ByVal UserURL As String) As ExchangeOptions```
This function should validate the user URL (```UserURL``` argument) and return an [ExchangeOptions](#exchangeoptions) object if it is your plugin user; otherwise, return a new empty ```ExchangeOptions```
If this is the user of your plugin, you must extract the UserName from the URL and return a new ```ExchangeOptions``` instance with the ```UserName``` and ```Site``` name.
If it is a channel, you must also set the ```IsChannel``` field (in ```ExchangeOptions```) to ```True```
For your convenience, this structure provides two default initializers. Use them so you don't forget the fields.
### IsMyImageVideo
```Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions```
Same as [```IsMyUser```](#ismyuser), but this function checks the ```URL``` argument to see if the ```URL``` is your site's media data. It is used for standalone video downloader.
Must return the same as the [```IsMyUser```](#ismyuser), but ```UserName``` can be **any non-empty value**.
### GetSpecialData
```Function GetSpecialData(ByVal URL As String) As IEnumerable(Of IPluginUserMedia)```
After ```IsMyImageVideo```, if this is your plugin's media file, this function will be called.
This function must return an IEnumerable ```IPluginUserMedia``` object. This object must contain the data URL and the file name (with extension). Any other fields are optional.
### GetInstance
```Function GetInstance(ByVal What As Download) As IPluginContentProvider```
**This is the required main function that returns a UserData instance!**
The ```What``` argument indicates who called this function.
- ```Main``` - user instance
- ```SavedPosts``` - instance of saved posts
- ```Channel``` - an instance of a channel (main window).
You must return an initialized instance of the ```IPluginContentProvider``` object.
### Load
```Sub Load(ByVal XMLValues As IEnumerable(Of KeyValuePair(Of String, String)))```
This function sends the XML fields of your site node to your settings class when the host is initialized.
### BeginInit
Called at the start of host initialization.
### EndInit
Called at the end of host initialization.
### BeginUpdate
Called from the settings root when the settings ```BeginUpdate``` function is called.
### EndUpdate
Called from the settings root when the settings ```EndUpdate``` function is called.
### Available
```Function Available(ByVal What As Download) As Boolean```
If your plugin needs to be checked for availability (you can see an example in the [```API.Reddit.SiteSettings```](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/Reddit/SiteSettings.vb)), place the check code here. Otherwise, this function **MUST** return ```true```.
If this function returns ```false```, users of your plugin site will not be downloaded.
### ReadyToDownload
```Function ReadyToDownload(ByVal What As Download) As Boolean```
**It is not the same as ```Available```.**
This function is an additional method for checking your class parameters. If your parameters don't need to be validated, just return true.
With this function, you can check the main settings parameters that authorization depends on (for example) or anything else your parser requires.
### DownloadStarted
```Sub DownloadStarted(ByVal What As Download)```
This function will be called when the parser **thread** is initialized. **Not for user parsing!**
### BeforeStartDownload
```Sub BeforeStartDownload(ByVal User As Object, ByVal What As Download)```
This function will be called before the user starts downloading. Here you can set some of user options you need.
### AfterDownload
```Sub AfterDownload(ByVal User As Object, ByVal What As Download)```
This function will be called after the user download is complete.
### DownloadDone
```Sub DownloadDone(ByVal What As Download)```
This function will be called when the parser **thread** is closed.
### OpenSettingsForm
If your site settings have a special settings form, this function will be called by clicking on the button in the site settings form.
**Your form must be opened with the ```ShowDialog``` method!**
### UserOptions
```Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)```
**If your site settings have special user options**, this function will be called by clicking on the ```Options``` button in the ```UserCreatorForm```.
```Options``` is an object parameter (**reference**). **If it is null, you must create a new instance of your option exchange class.** SCrawler does not know what class you are using to exchange. So SCrawler will exchange this class (as an object) between your settings class and user instance! So **this class object cannot be null**!
If the ```OpenForm``` argument = ```True```, your options form **must** be opened (using the ShowDialog method). This form must provide controls for change parameters obtained using the ```Options```.
## IPluginContentProvider
### ProgressChanged
```Event ProgressChanged(ByVal Count As Integer)```
Event for performing progress. When this event is raised, SCrawler downloading progress will be performed for this number (```Count``` (default = 1)).
### TotalCountChanged
```Event TotalCountChanged(ByVal Count As Integer)```
When you are starts downloading (before start), raise this event with the ```Count``` = total amount of media to download.
### Thrower
This is an exception thrower.
While debugging, you can use any class you want, but **SCrawler will replace that instance with my thrower**.
To work, you don't need to develop a special object of this interface.
Read more [here](#ithrower)
### LogProvider
This is an interface for sending exceptions and messages to the program log.
While debugging, you can use any class you want, but **SCrawler will replace that instance with my log provider**.
To work, you don't need to develop a special object of this interface.
Read more [here](#ilogprovider)
### Settings
This object will be replaced by SCrawler with the active settings instance. **ATTENTION! Don't instantiate settings!**
### Other properties
**All of these properties will be delivered using SCrawler!**
- ```Name``` - user name
- ```ID```
- ```ParseUserMediaOnly```
- ```UserDescription```
- ```ExistingContentList``` - collection of the existing downloaded data
- ```TempPostsList``` - collection of the existing downloaded post id
- ```TempMediaList``` - this is a list of new media to download. You **must** add data to this list!
- ```UserExists``` - indicates that the user exists on the site
- ```UserSuspended``` - indicates that the user's profile suspended (or blocked) on the site
- ```IsSavedPosts``` - this is the instance for downloading saved posts
- ```SeparateVideoFolder``` - if true, the videos **must** be saved separately from the images folder
- ```DataPath``` - download path
- ```PostsNumberLimit``` - the **maximum** number of posts a user has requested to download! This is a **nullable** argument. So, if it's ```Nothing``` (or ```null``` in C#), then no limited download is requested!
### ExchangeOptionsGet
```Function ExchangeOptionsGet() As Object```
This function is called by the UserCreatorForm when **editing** an existing user (not when creating a new one).
This function must return the ```ExchangeOptions``` (same as [```UserOptions```](#useroptions) function) with the values stored in this user instance.
### ExchangeOptionsSet
```Sub ExchangeOptionsSet(ByVal Obj As Object)```
Set options in this instance with ```ExchangeOptions```. ```Obj``` is the same object as in [```UserOptions```](#useroptions) function.
### XmlFieldsSet
```Sub XmlFieldsSet(ByVal Fields As List(Of KeyValuePair(Of String, String)))```
Delivery of XML fields stored in the user settings file, to the current instance.
- Key - XML field name
- Value - XML field value
### XmlFieldsGet
```Function XmlFieldsGet() As List(Of KeyValuePair(Of String, String))```
You must return the XML fields that you want to store in the user settings file.
- Key - XML field name
- Value - XML field value
### GetMedia
Parse site user data
### Download
Download data using parsed information
## IThrower
This is an exception thrower. If the user requested to cancel operation or delete user instance, an exception will be thrown. Use the ```IThrower.ThrowAny``` method in your code to stop executing when it's requested!
Exceptions:
- ```OperationCanceledException``` - when user requested to cancel operation
- ```ObjectDisposedException``` - when user deletes user instance
## ILogProvider
This is an interface for sending exceptions and messages to the program log.
Functions:
- ```Add(ByVal Message As String)``` - add ```Message``` to the program log
- ```Add```
- ```ex``` - your exception
- ```Message``` - additional message
- ```ShowMainMsg``` - show main message
- ```ShowErrorMsg``` - show error message
- ```SendInLog``` - message and error message will be sent to the program log
# Objects
## PropertyValue
This is the base object for interacting with SCrawler and bi-directional communication.
There are three initialization constructors:
- By initial ```value```. **If the value is ```null```, this constructor will throw an error!** Otherwise, the type will be extracted from the value.
- By ```value``` and ```type```.
- By ```value```, ```type``` and ```function```. If you want, you can delegate a function to be called when the value changes. You can see an example in the Instagram ```SiteSettings``` class
If your initial value is null, you **MUST** set the type.
**[Only these types are available](#propertyvalue-value-types)**
## IPropertyValue
**Don't use this interface. This interface is only compatible with SCrawler. Always use the ```PropertyValue``` class!**
## PropertyData
This is a structure for exchanging properties between classes without saving. It is currently used for [```PropertiesDataChecker```](#propertiesdatachecker) attribute.
- ```Name``` - property name
- ```Value``` - property value
## ExchangeOptions
For your convenience, this structure provides two default initializers. Use them so you don't forget the fields.
Currently used in [```IsMyUser```](#ismyuser) and [```IsMyImageVideo```](#ismyimagevideo)
- ```UserName``` - user name or any other data (if specified in the method description)
- ```SiteName``` - site name
- ```HostKey``` - aet automatically via ```SettingsHost```
- ```IsChannel``` - this user is a channel
# Attributes
## PropertyOption
This attribute allows you to add your property to the Scrawler site settings form. Only works with ```PropertyValue``` [object](#propertyvalue).
Options:
- ```PropertyName``` - automatically set when attribute is initialized. Corresponds to the property name.
- ```Type``` - property value type. Can be obtained automatically from ```PropertyValue```.
- ```ControlText``` - this text will be displayed on the control in the settings form.
- ```ControlToolTip``` - Control ToolTip.
- ```ThreeStates``` - CheckBox option. If ```true```, then the CheckBox will be displayed in three states: ```Checked```, ```Unchecked```, ```Indeterminate```. Default: ```false```.
- ```AllowNull``` - If ```false```, then the settings form will display an error message when trying to save the property value if the property value is ```null```. Default: ```true```.
- ```LeftOffset``` - Just a design option. This is just a control offset from the left border of the form (just for beauty).
- ```IsAuth``` - Default: ```false```. If at least one property has a ```PropertyOption``` with this parameter, then the controls in the settings form will be divided into two blocks: ```Authorization``` and ```Other```. Just a design option.
## PXML
This attribute specifies that your property should be added to the SCrawler settings XML file. When launched, SCrawler will replace the value of your property with the value from the XML settings file.
Again. Only works with ```PropertyValue``` [object](#propertyvalue).
## PropertyUpdater
This attribute provides a mechanism for updating a specific property. Attributed can only be applied to a method. Allows multiple definitions. Function **MUST** return a ```Boolean``` (```bool``` in C#) with ```true``` indicating that the value has been updated and ```false``` otherwise.
- ```UpdatingPropertyName``` - the name of the property to be updated
- ```Dependent``` - array of the property names on which update depends
## Manifest
This attribute provides the plugin key to associate users with the plugin. I recommend using the pattern: ```DeveloperName_Site```.
## SpecialForm
I have developed a fully integrated plugin environment. This means that any of your class property can be displayed in the SCrawler internal forms (for example, ```SettingsForm```, ```UserCreatorForm```). But if your plugin requires authorization or you want to manipulate your settings in a separate form, you can use the **```SpecialForm```** class attribute. This attribute allows multiple definitions for the same class and two working modes.
**```SpecialForm(True)```**, for class settings. This means that an additional button will be added to the site settings form to opening your settings form. In this case, it is necessary to develop an additional form that must be opened (using the ```ShowDialog``` method) by the ```OpenSettingsForm``` [class function](#opensettingsform).
**```SpecialForm(False)```**, for class settings. This means that the ```Options``` button will be enabled on the user creator form. In this case, you must develop an additional exchange class and form. This works with the ```UserOptions``` class function. This [function](#useroptions) has two arguments: ```Options``` (reference) and ```OpenForm```.
## Provider
This attribute indicates that the property is a provider. The provider must be an ```IFormatProvider```.
There are two working ways: check fields (```FieldsChecker```) and not.
```FieldsChecker``` = ```true```. Convert text from TextBox in settings form to value
```FieldsChecker``` = ```false```. Converting a value by a specific converter. Also used to convert a value from XML to an object.
## ControlNumber
Just a design attribute. Allows you to place controls on the settings form in a specific order. The number means the zero-based control position at the top of the form.
## PropertiesDataChecker
An attribute for the validation method. ```Names``` is an array of property names to check values. The method **must** be a function with an ```IEnumerable``` argument of [```PropertyData```](#propertydata) and ```boolean``` return.
Returning ```true``` means the values are OK.
If you need to check the values of some properties before saving the settings, use this attribute.
## SeparatedTasks
If your plugin needs to run on a separate thread add the ```SeparatedTasks``` class attribute. If you want to limit download tasks per your plugin, you can set number of tasks for this attribute. For example: ```SeparatedTasks(1)``` means that only one user will be downloaded per task; ```SeparatedTasks(2)``` means that two users will be downloaded at the same time for each task. The number argument is optional. This means that if you don't specify a tasks number, the global tasks number will be used instead. In additional, the ```TaskCounter``` property [attribute](#taskcounter) has the highest priority. This means that if you make a property (to allow users to change the number of tasks) with a ```TaskCounter``` [attribute](#taskcounter), the number of ```SeparatedTasks``` will be ignored. But the ```SeparatedTasks``` attribute **MUST** be added to the class anyway if you want to run your plugin on the separate thread.
## TaskCounter
If you want users to be able to control the number of download tasks in a thread, use this attribute. Only works with [```SeparatedTasks```](#separatedtasks)
## SavedPosts
If your plugin provides downloading of saved posts, add the ```SavedPosts``` class attribute.
## UseInternalDownloader
User data instance attribute (for [```IPluginContentProvider```](#iplugincontentprovider)). Means that after parsing, SCrawler should use the internal downloader.
## Github
Assembly attribute. Not used yet, but in future versions will allow SCrawler to check for updates to your plugin.
- ```Name``` - your GitHub username
- ```RepoName``` - your repository name
# PropertyValue value types
| vb.net | C# | NET |
| ---- | ---- | ---- |
| Boolean | bool | System.Boolean |
| Byte | byte | System.Byte |
| SByte | sbyte | System.SByte |
| Short | short | System.Int16 |
| UShort | ushort | System.UInt16 |
| Integer | int | System.Int32 |
| UInteger | uint | System.UInt32 |
| Long | long | System.Int64 |
| ULong | ulong | System.UInt64 |
| Double | double | System.Double |
| Single | float | System.Single |
| Decimal | decimal | System.Decimal |
| String | string | System.String |
| Date | DateTime | System.DateTime |
| TimeSpan | TimeSpan | System.TimeSpan |