Compare commits

...

64 Commits

Author SHA1 Message Date
Andy
e51debc027 2023.6.8.0
YT.Music: append artist name to music playlist output path
YT.MediaItem: fixed opening paths to downloaded playlists and channels
YT.YouTubeMediaContainerBase: save thumbnail path for playlist and channel
UserDataBase: remove old line of code
API.Twitter: fixed profile not fully downloaded
SiteEditorForm: corrected form size for small monitors
2023-06-08 17:27:19 +03:00
Andy
938042ea9e 2023.6.5.0
YT settings: removed property 'ItemsListLimit', add property 'ReplaceModificationDate'
YT.MediaItem: fix 'Pending'
YT.VideoListForm: add 'Shift' to add without downloading; add 'F5' hot key to start download; remove list items limit; fix item 'Pending', fixed items queue

UserDataBase: add 'IconBannerDownloaded' properties; add 'HOST.Available' check to 'DownloadSingleObject'; update file deletion in 'DownloadContentDefault'; add truncating '_TempPostsList' if number of ids > 1000
Instagram: add authorization headers
Mastodon: implement 'DownloadIconBanner'; update 'ReparseMissing' function
Reddit: implement 'DownloadIconBanner'
Twitter: implement 'DownloadIconBanner'; update parsers to parse posts with two videos; implement gallery-dl for all function; remove headers from settings
Download.DownloadProgress: remove main progress perform when downloading saved posts
VideoDownloaderForm: bind the 'BTT_ADD_URLS_ARR' button to the 'BTT_ADD_KeyClick' function
UsersInfoForm: add folder opening on double click on an item
ListImagesLoader: fix refill bug when the number of filtered profiles = 0
TrayIcon: add standalone downloader to context menu
DownloadableMediaHost: fix a bug when not downloaded videos do not appear in the list when loading the program
2023-06-05 19:36:35 +03:00
Andy
abdef81e5f Update README.md 2023-06-03 11:51:01 +03:00
Andy
e868c2e694 2023.5.12.0
IPluginContentProvider: add 'ProgressPreChanged' and 'ProgressPreMaximumChanged' events
YT.MediaItem: change folder opening on double click
YT.VideoListForm: change the icon for the 'Download' button

Add advanced progress
Add user metrics calculation
UserDataBase: fix GIF hash bug
Instagram: heic to jpg
Mastodon.SiteSettings: add the main domain to the list of domains with saving the settings
Mastodon.UserData: handle 'Forbidden' error; fix bug in parsing non-user posts
Pinterest: remove cookies requirement for saved posts
PornHub: fix resolutions issue; add 'DownloadUHD' option
Reddit: fix missing images bug; fix broken images bug; update container parsing function
MainFrame: fix collection pointing bug
2023-05-12 20:00:32 +03:00
Andy
b2a9b22478 2023.4.28.0
Plugins
IPluginContentProvider: added DownloadSingleObject function; added tokens to GetMedia and Download functions; removed GetSpecialData function
Add IDownloadableMedia interface
Removed 'Channel' option from all functions and enums
ISiteSettings: added GetSingleMediaInstance function
ExchangeOptions: removed 'IsChannel'
UserMediaTypes: added Audio and AudioPre enums
IUserMedia, PluginUserMedia: changed ContentType and DownloadState from integers to their enums

SCrawler
Add YouTube standalone downloader
Add gallery-dl & yt-dlp support
Remove 'UserInfo' requirement from 'ProfilesSaved'
Update 'SiteSettingsBase' to use domains and Netscape cookies
UserDataBase: remove channels; remove old 'Merge' const; standardize SavedPosts file naming; move 'ValidateMD5' function from Twitter to UserDataBase to use it in other UserData classes; add 'DownloadSingleObject' environment for single posts; add validating file extension for m3u8 during download; add reindex of video file during download

Rewritten DomainsContainer
Create a universal settings form and PSettingsArttribute
Gfycat, Imgur: turn these classes into IUserData to download a single object

All plugins: update 'GetInstance' function for saved posts; update domains where implemented; remove 'OptionForm' where it exists; update options where they exist; update unix date providers; reconfigure channels where they exist

LPSG: fix attachments; update converters and regex
Add sites: ThisVid, Mastodon, Pinterest, YouTube, YouTube music
Reddit: standardize container parsing for all data types; new channel environment; fix 'ReparseMissing' function; redirect data downloading to the base download function, saved crossposts support
Twitter: fixed gif path bug; fixed downloading saved posts
PornHub: hide unnecessary errors; photo galleries bug
RedGifs: add 'UserAgent' option

Added icons to download progress

Rename some objects
Completely redesigned standalone downloader form and rewritten its environment
WebClient2: update to use tokens

Labels: update label form (save labels to file only when OK button is clicked); change removing labels.txt from recycle bin to permanent; disable storing label 'NoParsedUser'

UserCreatorForm: remove the 'Channel' checkbox and related functions; ability to extract the user's URL from the buffer and apply parameters if found
Remove temporary 'EncryptCookies' module

MainFrame: added simplified way to create new users (Ctrl+Insert to create a new user with default parameters from clipboard URL); removed SCrawler command line argument "-v" (remove the ability to run SCrawler as video downloader)
PropertyValueHost: update for option forms compatibility
SettingsHost: removed 'GetSpecialData' fork; added 'GetSingleMediaInstance' fork
UserDataHost: update functions with tokens; update events; add 'DownloadSingleObject' function
Settings: add the ability to get environment from 4 destinations; add the ability to set the program environment manually; add CMDEncoding; add cache; remove the old function 'RemoveUnusedPlugins'; add 'STDownloader' properties; add YT compatibility; add new notification options; add deleting user settings file when 'SettingsCLS.Dispose()' if where are no users in SCrawler
UserFinder: remove old 'Merge' const; remove channel option
UserInfo: remove channel option
2023-04-28 10:13:46 +03:00
Andy
db9e2cfb88 2023.3.5.0
Fixed SavedPosts new file naming method
Fixed Twitter MD5 comparison error
Fixed ffmpeg file parts concatenation algorithm
2023-03-05 06:12:08 +03:00
Andy
85d8df96ca 2023.3.1.0
Add 'Path' plugin
UserDataBase: changed file names for saved posts; removed 'Self' property; add 'MyFileSettings' field; added UserSiteName; changed download envir algo
Twitter: added MD5 comparison; duplicate images removal option; UserSiteName parsing; download icon and banner
Instagram: added a new option for token 'www_claim'; removed requirement of token 'www_claim'; UserSiteName parsing; download icon
Reddit: UserSiteName parsing; download icon and banner
PornHub: fixed unicode titles
XHamster: added channels
ffmpeg: fixed max input length error during files combining; fixed encoding issue
Feed: added images centering; added BackColor and ForeColor change
MainFrame: added BackColor, ForeColor, and BackgroungImage change; added 'UpdateLogButton' when load completed
ListImagesLoader: fixed wrong notification when no users found
SettingsCLS: updated users loading algo
2023-03-01 20:35:52 +03:00
Andy
6ca90f0489 Update names 2023-02-25 10:19:18 +03:00
Andy
2a780a3acf Update README.md 2023-02-13 11:44:00 +03:00
Andy
290b5c4586 2023.2.5.0
Update to library environment
GlobalSettings: add UserAgent
Gfycat: fix RedGifs urls issue
2023-02-05 20:58:02 +03:00
Andy
f5e68a7032 Update Changelog.md 2023-01-28 05:50:07 +03:00
Andy
ece573dd40 Update README.md 2023-01-28 05:48:54 +03:00
Andy
1f1148020c 2023.1.27.0
Plugins: added 'Interaction' to 'Provider' attribute; added 'IPropertyProvider' interface
Hosts: update classes to work with new options
Instagram: fixed pinned post reload
Twitter: advanced options for GIFs
UserCreatorForm: change icon based on the selected site
UserSearchForm: change search function
2023-01-27 16:43:57 +03:00
Andy
fc226d549a 2023.1.24.1
Some Imgur albums won't download
Added icon for standalone downloader
2023-01-24 16:13:46 +03:00
Andy
602771d982 2023.1.24.0
Imgur albums not downloading
Collections: users in the collection are not banned
2023-01-24 06:05:40 +03:00
Andy
3e472b4f2b Update HowToSupport.md 2023-01-13 00:21:48 +03:00
Andy
30c3fe3b68 Update info
Update info
2023-01-12 07:38:17 +03:00
Andy
38c81b7a0b 2022.1.2.0
Redgifs: added token refresh interval; reduced interval value
Updated labels collection
PornHub: fixed bugs
Notifications: pressing any button opens SCrawler
User list loader finished
2023-01-02 18:53:24 +03:00
Andy
0fb6add751 Update UserData.vb 2022-12-27 15:19:40 +03:00
Andy
5d64b8c7ce 2022.12.27.0
XVideos: added 'Quickies'; fixed downloading.
Instagram: added more enable/disable options.
2022-12-27 15:04:56 +03:00
Andy
aabf6d62ab 2022.12.26.0
UserMedia: fixed plugin bugs
Instagram: updated algo and settings; update responser settings based on site response
PornHub: fixed bug in SiteSettings; fixed typos
RedGis: fixed downloading user profiles
XVideos: fixed user profile opening
UserDataBind: fixed multiple collection removing issue
DownloadedInfoForm: fixed user focusing
UserCreatorForm: add user name to form header if user exists
ListImagesLoader: changed loading algo
MainFrame: added channels button to tray context menu
Added ffmpeg fox x86
Fixed typos
2022-12-26 17:37:25 +03:00
Andy
03487185c5 Update names
Updated library objects
2022-12-24 15:45:12 +03:00
Andy
f0686bbc8e Fixes
Fixed typo
Added label icon to context menu
Fixed PornHub bug
Added 'Object' to IUserMedia
2022-12-24 15:18:04 +03:00
Andy
bdc7321331 2022.11.16.0
Add sites: PornHub, XHamster
Add saved xvideos posts downloading
PluginProvider: added TaskGroup attribute; added IUserMedia inteface; changed PluginUserMedia to IUserMedia in interface declarations; changed 'User' String to IPluginContentProvider in ISiteSettings sinterface
Added update the 'LOG' button at the end of the ProfileSaved download function
API.Base: added 'IUserMedia' compatibility for 'UserMedia'; moved 'GetImage' from 'UserPost' to 'ChannelsViewForm'; update constants in UserDataBase; updated UserDataBase to new UserInfo environment.
API.Instagram.UserData: fixed date issue
API.Reddit.SiteSettings: update user patterns
API.Twitter.Declarations: moved provider here from MainFrame
UserDataBind: updated to new UserInfo environment
ActiveDownloadingProgress: updated form rendering
AutoDownloader: added SpecialDelay
TDownloader: added 'Suspended' option; updated for TaskGroups
CollectionEditorForm: fixed order bug
LabelsForm: remove old stuff
UserEditorForm: added collection editing
MainFrame: improve label selection
Add import users
Added the ability to create a virtual collection and add a virtual user to a real collection
SettingsCLS: improve users loading
2022-11-16 13:41:45 +03:00
Andy
7d169acebc 2022.10.23.0
PluginProvider: added 'DoNotUse' attribute.
Channels: copying a channel to the 'ChannelsDeleted' folder before deleting.
Twitter: updated status codes.
AutoDownloader: removed base parameters initialization; updated 'ToEContainer' function; 'StartupDelay' default value = 1; added IIndexable; fixed NextExecutionDate; added task delay based on index and tasks count; updated user selections algorithms.
MainFrame: coloring the button 'Download All' depending on the pause; updated user selections algorithms.
DownloadGroups: added LabelsExcluded, Sites and SitesExcluded; updated initialization; updated ToEContainer function; updated user selections algorithms
IGroup, GroupParameters: added LabelsExcluded, Sites and SitesExcluded; added Import and Export functions
Removed TrayIcon notifications. All notifications are now ToastNotifications.
SettingsHost: 'DoNotUse' attribute
Settings: added 'GetUsers' predicate function
2022-10-23 16:39:01 +03:00
Andy
f5c156b8e5 2022.10.18.0
Moved UserMedia xml initialization to the structure itself
Added download with feed skip
Added silent mode (temporary disabling notifications)
Added additional Instagram protection
Excluding users whose profiles do not exist from downloading with groups and AutoDownloader
Feed: delete file bugs; reorder data after file deletion; video playback bugs
SiteSettingsForm: enable 'OK' button when editing cookies
Fixed collection users ban
Settings: disabling ffmpeg missing notification; advanced notification management
Added 'ToolStripKeyMenuItem' control
Plugins: deprecated XVIDEOS and LPSG plugin libraries; moved them to SCrawler.
Updated license
PluginProvider: added 'BeginEdit' and 'EndEdit' function to ISiteSettings; changed GetSpecialData (ISiteSettings) return type to IEnumerable
PluginsEnvironment: removed 'IsMyClass' attribute
MainFrame: grouped all download buttons into one menu; reorganized code; removed 'F2' hotkey
AutoDownloader: added advanced pause options; added buttons to tray icon and AutoDownloader form
MissingPosts: finished; activated functions that were disabled; added download functions to UserData classes
UserDataBase: ability to use responser; ability to download m3u8; extended 'DownloadingException' with optional argument 'EObj'; user index in collection (button tag) changed to user instance; extended information with user labels; updated 'ProcessException' function
Replaced download buttons with 'KeyClick' control
Replaced FDatePickerForm with my library's form
Collections: Deleting multiple collections - disabled confirmation; ban each user in collection
2022-10-18 12:05:31 +03:00
Andy
d91ee72eaa Update LICENSE 2022-10-18 12:02:43 +03:00
Andy
4e9de23b60 Deprecated
Moved to SCrawler
2022-10-18 11:54:50 +03:00
Andy
129558c262 2022.9.24.0
Fixed wrong image opening in Autodownloader
Fixed incorrect feed grid resizing when removing media
Fixed incorrect removal of users from the collection
Fixed Instagram function displaying number of requests: wrong value type.
Fixed XVIDEOS cycle bug
Collection: add multiple users
Collection: new collections at the top
Copying user data
Feed: 'Season' and 'Date' to the post title.
2022-09-24 20:26:40 +03:00
Andy
a3e79eb4bc Update UserDataBase.vb
Fixed typo
2022-09-17 20:28:27 +03:00
Andy
eb28255de3 2022.9.17.0
Extended filters by date
Added download by dates for multiple users
Changed validation of dates ranges in UserDataBase
Add user filters by dates
Add disabling site downloading
Fixed Twitter date validator
2022-09-17 19:59:55 +03:00
Andy
92be0994ae 2022.9.16.0
Removed some compatible functions
Fixed Settings.GetUser bug
Design improvements
Changed UserMediD comparer
FeedVideo design updated, incorrect time position fixed, bugs fixed
Fixed getting Reddit channel video thumbnail
2022-09-16 19:41:24 +03:00
Andy
9567b0a367 2022.9.13.0
Added video duration to the feed
Added skipping of pinned Instagram posts if they are already downloaded
2022-09-13 16:20:07 +03:00
Andy
c28c0e1ba3 2022.9.10.0
Fixed: missed posts are not saved
Fixed memory leaking because of the video
2022-09-10 12:28:40 +03:00
Andy
86771eee94 2022.9.8.1
Fixed unexpected memory leak when using the 'Feed' form
2022-09-08 22:24:36 +03:00
Andy
02e8a15ae3 2022.9.8.0
Temporary disabled RedGifs downloading
Added 'missing posts', 'feed'
Fixed minor bugs
2022-09-08 12:36:25 +03:00
Andy
443ab329d5 2022.8.28.0
Changed target platforms
Added RedGifs pics
Fixed Switcher limit bug
2022-08-28 04:08:54 +03:00
Andy
a16bb8de90 Update CONTRIBUTING.md 2022-08-26 20:38:33 +03:00
Andy
0af5e6f8d4 Update README.md 2022-08-26 20:37:54 +03:00
Andy
54ffe10f71 2022.8.22.0
Cleaned up the code
Replace some old functions with new ones
Adapted to the new library environment
Enable/Disable display user/downloaded image
Autodownloader option 'Show notification' not saved
Separate thread for standalone video downloader
Expanded the description of some errors with additional information
Fixed date/time renaming issue
Fixed internal library bugs
Fixed minor bugs
2022-08-22 02:42:36 +03:00
Andy
e0dc66e0da 2022.7.7.0
Brushed the code in some classes
Extended PropertyOption attribute
Removed AuthNullException
Moved ExitException to UserData class
Removed Instagram HashUpdateRequired and its environment
Changed Reddit response status code check
Twitter images bug
Added Scheduler, task startup delay, webp to jpg
Fixed Stop button bug
Minor changes
2022-07-07 14:11:18 +03:00
Andy
ab020d9b5f 2022.6.10.0
Instagram User ID
2022-06-10 21:13:35 +03:00
Andy
4ba1624edf Update MainWindowGroups.png 2022-06-09 07:55:58 +03:00
Andy
f3d956f33f 2022.6.7.0
Fixed some design issues
2022-06-07 20:00:36 +03:00
Andy
4a5e050201 Update README.md 2022-06-06 21:49:00 +03:00
Andy
dd272c6f6d 2022.6.6.0
Minor fixes
Fixed Twitter gifs
2022-06-06 21:32:43 +03:00
Andy
fbcda1ae75 2022.6.4.0
Added pause automation
Extended automation information
Updated automation checker
2022-06-04 02:43:46 +03:00
Andy
9e87369c9b Update ReadMe 2022-06-04 02:14:18 +03:00
Andy
cc3618a50e Update CONTRIBUTING.md 2022-06-04 02:10:44 +03:00
Andy
33b9e9cfc6 2022.6.3.0
Updated plugin environments and dependencies
Added automation
Fixed Insta hash issue
Updated groups
Added toast notifications
Updated tagged posts notifications
Updated M3U8; fixed audio issue
Extended some of log exceptions
Fixed minor bugs
Other minor improvements
2022-06-03 20:42:28 +03:00
Andy
26dca2246e Update README.md 2022-05-27 21:40:30 +03:00
Andy
60b459e217 3.0.0.10
Added downloading groups
Added downloading Twitter saved posts
Added scripts when closing and completing the download
Opening Info and Progress forms when downloads start
Disabling the opening of forms Info and Progress at the start of downloads if it was once closed
Added focusing the main window when opening Info or Progress forms
Fixed downloading Instagram tagged data
Fixed forbidden characters Instagram stories
Updated form field checkers
Fixed downloading Imgur and Gfycat if they were posted on Reddit
Fixed separate Instagram posts were not downloading via the Video Downloader form.
Date time filenames
Twitter 4K images
2022-05-23 15:51:08 +03:00
Andy
f491e03812 Merge pull request #37 from unknown81311/main
Update MainFrame.vb
2022-05-02 08:11:35 +03:00
unknown81311
418f44edfd Update MainFrame.vb 2022-04-27 19:24:37 -06:00
Andy
075a2b9b80 3.0.0.9
Updated labels class
Moved some settings from SettingsCLS to LabelsKeeper
Excluded labels
Disable user grouping
Show groups of user sites when filtering by labels
Removed adding 'No Parsed' internal label
Fixed redownloading Instagram Stories
Changed global settings form
Updated Labels form
Fixed text separator in UserCreatorForm
Add target user if hidden
2022-04-24 20:35:30 +03:00
Andy
20c74ec8f1 Update ProgramsComparison.md 2022-04-20 18:48:06 +03:00
Andy
0594e77e0b 3.0.0.8
Script mode command
Disabled Instagram error 403
Fixed script does not run
2022-04-19 14:58:56 +03:00
Andy
a5fa935e76 3.0.0.7
Added script usage
Fixed  downloading of LPSG images
Fixed Instagram Stories
Fixed date/time file pattern
2022-04-14 18:12:01 +03:00
Andy
c90dd5637e Update Plugins.md 2022-04-12 05:41:03 +03:00
Andy
9a301ebc5e 3.0.0.6
Added GoTo Start/End channels buttons
Fixed saved Reddit posts downloading
Fixed Reddit accessibility check
Disabled main progress bar progress when downloading saved posts
Added Date and Time for Stories and Tagged Photos
2022-04-04 03:00:22 +03:00
Andy
11a590f14e 3.0.0.5
Added options for Reddit channel and user
Fixed minor bugs and typos
2022-04-02 04:36:38 +03:00
Andy
975d46715c 3.0.0.4
Changed XVIDEOS.M3U8 errors
Added TryCatch to XVIDEOS.UserData.Download
Removed old declarations in UserDataBase
Changed description replacement in UserDataHost
2022-03-26 20:00:55 +03:00
Andy
726fc486ce 3.0.0.3
Added additional 'download all' options
Fixed ListImagesLoader (User.FitToAddParams)
Fixed Instagram default value of SleepTimerOnPostsLimit
Fixed XVIDEOS typo
2022-03-24 19:14:58 +03:00
Andy
ede81f9d05 Update HowToSupport.md 2022-03-23 00:41:18 +03:00
432 changed files with 40134 additions and 12732 deletions

3
.gitattributes vendored
View File

@@ -1,2 +1,5 @@
# Auto detect text files and perform LF normalization
* text=auto
# Declare files that will always have CRLF line endings on checkout.
*.md text eol=crlf

4
.github/FUNDING.yml vendored
View File

@@ -3,11 +3,11 @@
github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
patreon: # Replace with a single Patreon username
open_collective: # Replace with a single Open Collective username
ko_fi: andyprogram
ko_fi: #andyprogram
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
otechie: # Replace with a single Otechie username
lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
custom: ['https://blockchair.com/bitcoin/address/BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET']
custom: ['https://github.com/AAndyProgram/SCrawler/blob/main/HowToSupport.md']

View File

@@ -12,12 +12,17 @@ A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Profile URL:
2. Do something
3. See error
1. **Profile URL**:
2. **Post URL**:
3. Do something
4. See error
**Log data**
If the program log contains any data.
<details>
<summary>Log data</summary>
<pre>
If the program log contains any data, replace this line with the log data. If the program log does not contain any data, write here about.
</pre>
</details>
**Expected behavior**
A clear and concise description of what you expected to happen.
@@ -26,8 +31,13 @@ A clear and concise description of what you expected to happen.
If applicable, add screenshots to help explain your problem.
**Release information (please complete the following information):**
- Architecture [e.g. x86, x64]
- Version [e.g. 2.0.0.0]
- OS: [e.g. Windows 10, Windows 11]
- Architecture: [e.g. x86, x64]
- Version: [e.g. 2023.3.5.0]
- NET.Framework version:
- ffmpeg version (command `ffmpeg -version`):
- yt-dlp version (command `yt-dlp --version`):
- gallery-dl version (command `gallery-dl --version`):
**Additional context**
Add any other context about the problem here.

View File

@@ -1,6 +1,6 @@
---
name: Add plugin
about: Add plugin to plugin list
name: I developed a plugin for SCrawler
about: I developed a plugin for SCrawler. Add plugin to plugin list.
title: "[NEW PLUGIN]"
labels: 'New Plugin'
assignees: ''

4
.gitignore vendored
View File

@@ -9,8 +9,9 @@
*.user
*.userosscache
*.sln.docstates
ffmpeg.exe
.obsidian/
ToDo.txt
ToDo.md
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
@@ -33,7 +34,6 @@ bld/
[Oo]bj/
[Ll]og/
[Ll]ogs/
ffmpeg/
Info/
# Visual Studio 2015/2017 cache/options directory

View File

@@ -3,39 +3,41 @@
I welcome requests! Follow these steps to contribute:
1. Find an [issue](https://github.com/AAndyProgram/SCrawler/issues) that needs assistance.
2. Let me know you are working on it by posting a comment on the issue.
3. If you find an error in the code, please provide a link to the file and the line number.
4. If you have a suggestion to change the code, you can post a block of code to replace. I don't currently have time to learn pull requests, so it might work this way.
1. Let me know you are working on it by posting a comment on the issue.
1. If you find an error in the code, please provide a link to the file and the line number.
1. If you have a code change suggestion, you can post a replacement code block. I also accept pull requests.
# How to build from source
1. Delete the `PersonalUtilities` project from the solution.
1. Delete the `PersonalUtilities.Notifications` project from the solution.
1. The following libraries must be added to project references with the '**Copy to output folder**' option:
- `PersonalUtilities.dll`
- `PersonalUtilities.Notifications.dll`
- `Microsoft.Toolkit.Uwp.Notifications.dll`
- `System.ValueTuple.dll`
1. Import `PersonalUtilities.Functions` for the whole project.
1. Delete the "PersonalUtilities" project from the solution.
2. Add the latest version of the "PersonalUtilities.dll" library (from the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest)).
3. Import PersonalUtilities.Functions for the whole project.
**Always use the correct "PersonalUtilities.dll" library. You must download this library from the release of the code you downloaded.**
**Always use the correct libraries. You must download libraries from the same release date as the code commit date.**
# How to request a new site
**I'm currently not accepting requests to develop new sites.**
1. Check [issues](https://github.com/AAndyProgram/SCrawler/issues) (open and [closed](https://github.com/AAndyProgram/SCrawler/issues?q=is%3Aissue+is%3Aclosed)) and [discussions](https://github.com/AAndyProgram/SCrawler/discussions) to find your issue. Perhaps I have already answered your request.
2. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
- If I'm interested in a site you want to add, it may be added in future releases.
- If the site has an API that does not require authorization, it may be added in the coming releases.
- You can make it faster by posting a link to the API. **I don't use OAuth authentication** in my application, so if it's not too hard to make a new parsing algorithm **without OAuth** authorization, I can start developing it in the coming days. Otherwise, I need time to figure out how to do it.
- If the site does not have an API that does not require authorization, this may take some time.
- If you will be posting request urls **without OAuth** authentication, I might consider adding your site if I have time.
- If I'm **not** interested in the site you want to add, you can pay to have it added by making a donation of approximately $10. **But before that, you still need to create an issue. If I'm not interested, you can offer me a deal to develop it for money. I'll check the site you want to add, check the availability of the API and tell you how much time I need to develop it and the price. If you agree, I will do it.** [![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
1. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
# Requirements for new site requests
**Attention! I'll add a new site only if I'm interested. I also have a life, and any development takes time.**
- Post a link to the site's API
- Post request URLs **without OAuth** authentication
- Post a **complete cURL** request which provides the required information (JSON is better)
**I don't use OAuth authentication** in my application, so if it's not too hard to make a new parsing algorithm **without OAuth** authorization, I can start developing it in the coming days. Otherwise, I need time to figure out how to do it.
If I'm interested in a site you want to add, it may be added in future releases.
# Sites I will never develop
- Facebook
# Sites requested by users
- TikTok
- API for receiving data without authorization was not found. Therefore, I don't have time to start developing this site parsing algorithm. If anyone knows of requests that may collect data without OAuth authentication, please let me know.
# Contact me
[![matrix](https://img.shields.io/badge/Matrix-%40andyprogram%3Amatrix.org-informational)](https://matrix.to/#/@andyprogram:matrix.org)
- Tumblr

View File

@@ -1,5 +1,558 @@
# 2023.6.8.0
*2023-06-08*
- Added
- YouTube: append artist name to music playlist output path
- YouTube: save thumbnail path for playlist and channel
- Fixed
- YouTube: opening paths to downloaded playlists and channels
- Twitter: profile not fully downloaded
- Corrected form size for small monitors (Issue #136)
# 2023.6.5.0
*2023-06-05*
- Added
- **Instagram**: add additional authorization headers
- Setting to prevent user icon and banner from downloading (Request #129)
- Add standalone downloader to tray context menu
- YouTube downloader: added `Replace modification date` property
- Minor improvements
- Fixed
- Fascist **Twitter**: posts not downloading (new API)
- Main window: refill bug when the number of filtered profiles = 0
- Standalone downloader: new items are not added to the queue
- Standalone downloader: bug when not downloaded videos do not appear in the list when loading the program
- Standalone downloader: add videos array not working
- Saved posts: remove main progress perform when downloading saved posts
- Minor bugs
# 2023.5.12.0
*2023-05-12*
- Added
- Advanced progress (make progress bars more informative)
- User metrics calculation
- Reddit: improve parsing function
- PornHub: add `Download UHD` option
- Fixed
- MD5 GIF hash bug
- Mastodon: handle 'Forbidden' error
- Mastodon: bug in parsing non-user posts
- Pinterest: remove cookies requirement for saved posts
- PornHub: resolutions issue
- Reddit: missing & broken images bug
- Main window: collection pointing bug
# 2023.4.28.0
*2023-04-28*
- Added
- **YouTube**
- **YouTube Music**
- **Mastodon**
- **Pinterest**
- **ThisVid**
- **YouTube downloader (standalone app)**
- Redesigned standalone downloader and update environment
- Added icons to download progress
- Added icons to saved posts downloader
- **Cookies**: new ways to add cookies. You can now export cookies using the browser extension and then import them into SCrawler!
- User creation: ability to extract the user's URL from the buffer and apply parameters if found
- User creation: simplified way to create new users (`Ctrl+Insert` to create a new user with default parameters from clipboard URL)
- Ability to customize the placement of ffmpeg (and other) files
- Ability to customize the command line encoding
- New notification options for standalone downloader
- Reddit: now it can download saved crossposts
- RedGifs: added `UserAgent` option
- Other improvements
- Removed
- User creation: remove the 'Channel' checkbox because it confuses people
- Removed an ability to open SCrawler with `-v` argument
- All ways to create users except URL. You can only properly create a user using the user's URL.
- Plugins
- Added `IDownloadableMedia` interface
- Removed `Channel` option from all functions and enums
- ISiteSettings: added `GetSingleMediaInstance` function
- IPluginContentProvider: added `DownloadSingleObject` function
- IPluginContentProvider: added tokens to `GetMedia` and `Download` functions
- IPluginContentProvider: removed `GetSpecialData` function
- UserMediaTypes: added `Audio` and `AudioPre` enums
- Fixed
- LPSG: attachments not downloading (Issue #114)
- Twitter: saved posts not downloading (Issue #119)
- XVIDEOS: saved posts not downloading
- Deleting labels file
- PornHub: hide unnecessary errors (Issue #116)
- PornHub: photo galleries bug (Issue #115)
- Minor bugs
# 2023.3.5.0
*2023-03-05*
- Fixed
- A bug in the new way of naming `SavedPosts` data files.
- An error that could occur during Twitter MD5 comparison.
- A bug in the ffmpeg file parts concatenation algorithm that could occur in some cases.
# 2023.3.1.0
*2023-03-01*
- Added
- **Path plugin.** Now you can add paths. *This may be suitable if you want to add a collection of media data to a specific user collection.*
- MainWindow: setting a background image
- MainWindow: setting background color and font color
- Feed: setting background color and font color
- Feed: (Request #108) center the image in the feed grid
- Users: the ability to use user site name (if it exists) as a friendly name (on supported sites: Reddit, Twitter, Instagram)
- Users: the ability to update user site name every time
- Twitter: ability to download images using MD5 comparison to protect against duplicate downloads *(this may be suitable for the users who post the same image many times)*
- Twitter: one-time duplicate image removal option
- XHamster: (Request #107) added channels downloading
- Updated
- Updated ffmpeg to version [5.1.2](https://github.com/GyanD/codexffmpeg/releases/tag/5.1.2)
- Fixed
- PornHub: (Issue #106) unicode titles
- (Issue #106) problem with non-Latin characters
- ffmpeg: maximum input length error when merging parts of files
# 2023.2.5.0
*2023-02-05*
- Added
- The ability to configure UserAgent
- Fixed
- (Issue #101) Failed download Gfycat video in some cases
# 2023.1.27.0
*2023-01-27*
- Added
- Advanced Twitter options for GIFs
- Changing the icon of the user creation form based on the selected site
- Fixed
- Pinned Instagram posts reload every time
- Plugins
- Added
- `Interaction` option to the `Provider` attribute
- `IPropertyProvider` interface
# 2023.1.24.1
*2023-01-24*
- Added
- Icon for standalone downloader
- Fixed
- (Issue #100) some Imgur albums won't download
# 2023.1.24.0
*2023-01-24*
- Fixed
- (Issue #100) Imgur albums not downloading
- When deleting a collection with the 'ban' option, users in the collection are not banned
# 2023.1.2.0
*2023-01-02*
- Added
- RedGifs: an ability to customize token refresh interval
- RedGifs: token refresh interval changed from 24 hours to 12 hours
- Updated labels collection
- Fixed
- PornHub: bug in the downloader
- PornHub: download additional non-user videos
- Reddit: bug in standalone downloader
- Fixed a bug in the user list loading algorithm
- Notifications: pressing any button opens SCrawler
# 2022.12.27.0
*2022-12-27*
- Added
- XVideos: added downloading 'Quickies'
- Instagram: added more enable/disable options
- Fixed
- XVideos not downloading (sorry, I broke it in a previous release)
# 2022.12.26.0
*2022-12-26*
**ATTENTION!**
**Instagram requirements changed. Headers and cookies are now required to download Timeline, Stories and Saved posts; hash to download tagged posts. Please update your credentials.**
**Instagram tagged posts no longer provide the total amount of tagged posts. I've corrected the tagged posts notification, but now I can't tell how many requests will be spent on downloading tagged posts. And from now on, one request will be spent on downloading each tagged post, because Instagram doesn't provide complete information about the tagged post with the site's response. In this case, if the number of tagged posts is 1000, 1000 requests will be spent. Be careful when downloading them. I highly recommend that you forcefully disable the downloading of tagged posts for a while.**
- Added
- Updated user loading algorithm
- Channels button to tray context menu
- (Request #96) Add FFmpeg to x86 version
- Fixed
- PornHub wrong behavior when downloading images
- Unable open XVideos user profile
- Cannot delete multiple collections at once
- Can't focus user from the download info form
- Instagram downloader not working
- (Issue #69) **RedGifs data is not downloading**. Again.
- Minor bugs
# 2022.11.16.0
*2022-11-16*
**ATTENTION! This version makes changes to the base SCrawler user configuration file. Since you started using this version, you still can downgrade. BUT! Once you add a virtual collection or a virtual user to a collection, you won't be able to downgrade without losing data.**
- Added
- **PornHub**
- **XHamster**
- An ability to download saved XVIDEOS posts
- Download indicator. While downloading, the rainbow tray icon changed to a blue arrow.
- Collections: the ability to edit a collection using a form
- Collections: the ability to create a **`virtual collection`** and add a **`virtual user`** to a real collection
- Collections: an easier way to added users to a collection
- Collections: an easier way to create collections
- Added icons for channels form context menu buttons
- More convenient change of user labels from the context menu of the user list
- Notifications: complete transition from default notifications to ToastNotifications
- Notifications: when you click on the notification that some of the channels are downloaded, the channels form opens
- Notifications: when you click on the notification that all users are downloaded, the main window form opens
- Notifications: when you click on the notification that the saved posts are downloaded, the saved posts form opens
- Import users
- Minor improvements
- Plugins
- Added
- `TaskGroup` attribute
- `IUserMedia` interface
- Changed
- `GetUserUrl` and `GetUserPostUrl` functions: `String UserName` and `String UserID` changed to ` IPluginContentProvider User`
- Fixed
- Collections editor: new added collections are still not added to the top of the collections list
- Users search form doesn't remember last size
- Minor bugs
# 2022.10.23.0
*2022-10-23*
- Added
- RedGifs token Auto-Renewal
- Download groups: ability to select sites
- Download groups: ability to exclude labels and sites
- AutoDownloader: ability to exclude labels and sites in ```All```, ```Default``` and ```Specified``` modes
- The ```Download All``` button turns blue when pause is enabled
- Updated Twitter status codes
- Minor improvements
- Fixed
- Updated Twitter status codes
- AutoDownloader: incorrect next run date in scheduler task information
- AutoDownloader: minor bugs
- (Issue #69) **RedGifs data is not downloading**. Requires token.
- Minor bugs
# 2022.10.18.0
*2022-10-18*
- Added
- **TikTok** ([limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
- **Search form** (```Ctrl+F```)
- Feed improvements
- Ability to save the download session for viewing later
- Ability to download user, excluding from the feed (use the ```Ctrl``` key with a button click of with a hot key press)
- Ability to disable the notification about the absence of the ffmpeg.exe file
- Extended user information with labels
- Advanced AutoDownloader pause options
- Added pause buttons to tray icon and AutoDownloader form
- Additional Instagram protection
- Advanced notification management
- Silent mode (temporarily disable notification)
- Excluding users whose profiles do not exist from downloading with groups and AutoDownloader
- Minor improvements
- Updated
- Grouped all download buttons into one menu
- **Finished missing posts**. You can now download missing posts if they exist.
- PluginProvider: added ```BeginEdit``` and ```EndEdit``` methods
- PluginProvider: ```GetSpecialData``` return type changed from ```IEnumerable(Of PluginUserMedia)``` to ```IEnumerable```
- XVIDEOS and LPSG plugins are moved from libraries to SCrawler
- Fixed
- (Issue #69) **RedGifs data is not downloading**. Requires cookies and token.
- Some minor bugs when deleting a collection
- Feed: start video playing may cause the program to freeze (strange behavior of the vlc library)
- Feed: videos hosted on Reddit not showing up in feed
- Feed: minor bugs
- Collection users were not banned when deleted with the ban option
- When trying to delete multiple collections, each collection asked for confirmation to delete
- Minor bugs
# 2022.9.24.0
*2022-09-24*
- Added
- Ability to copy user data to another destination
- Ability to add 'Session' and 'Date' values to the post title in the feed
- Minor feed improvements
- The newly created collection will now appear at the top of the list (after reopening the form)
- Ability to add multiple users at a time to the collection.
- Fixed
- Autodownloader opens a compressed image instead of a full one
- Incorrect resizing of the feed grid after deleting a media file
- Incorrect behavior when deleting/removing a user from a collection.
- An incorrect function that displayed the number of spent Instagram requests.
- Bug in the XVIDEOS downloader
- Minor bugs
# 2022.9.17.0
*2022-09-17*
- Added
- Added two date filters to filter users (in range, not in range)
- (Request #71) Download data for a specific date range
- The ability to disable site downloading (in the site settings form)
- Updated
- Plugins
- Fixed
- (Issue #71) ```Download data to the date``` doesn't work for Twitter
- Download data for a specific date range doesn't work for multiple users
- Incorrect feed sorting algorithm
- Minor bugs
# 2022.9.16.0
*2022-09-16*
- Fixed
- Failed to get video thumbnail for channel video post
- Incorrect rendering of the 'Feed' table when the number of columns is more than one
- Minor design bugs
# 2022.9.13.0
*2022-09-13*
- Added
- Video duration to the feed
- Fixed
- (Issue #70) Instagram posts not downloading if there are pinned posts that have already been downloaded
- Minor bugs
# 2022.9.10.0
*2022-09-10*
- Fixed
- The memory is still leaking. This time because of the video. *Using WMP was not the best choice.*
# 2022.9.8.1
*2022-09-08*
- Fixed
- Unexpected memory leak when using the 'Feed' form
# 2022.9.8.0
*2022-09-08*
- Added
- **Feed** (feed of downloaded media files)
- Missing posts tracking and management
- Simple scheduler notifications
- Fixed
- (Issue #67) Saved Instagram posts not downloading
# 2022.8.28.0
*2022-08-28*
- Added
- RedGifs icon
- Fixed
- Incorrect number of posts displayed in the Reddit channels downloader.
# 2022.8.22.0
*2022-08-22*
- Added
- Ability to enable/disable the display of the downloaded image in toast notifications (AutoDownloader)
- Ability to enable/disable the display of the user icon in toast notifications (AutoDownloader)
- Downloading with standalone video downloader has been moved to a separate thread
- Fixed
- (Issue #35) The file name does not change only by date
- (Issue #62) Internal library error
- AutoDownloader option ```Show notifications``` not saved
- Minor bugs
# 2022.7.7.0
*2022-07-07*
- Added
- **Scheduler** (creating multiple automation tasks)
- Automation startup delay
- Download ```webp``` in ```jpg``` format
- Development: the ability to create a label control, that provides some information
- Removed
- Instagram auto-fill hash from cookies
- Updated
- Plugins
- Fixed
- ```Stop``` option not working properly
- In some cases, Twitter image is not downloading
- Minor bugs
# 2022.6.10.0
*2022-06-10*
**Attention! From now on, Instagram requires Cookies, Hash and authorization headers!**
- Fixed
- Can't get Instagram user ID
# 2022.6.6.0
*2022-06-06*
- Added
- Ability to pause automation
- Fixed
- GIFs from Twitter not downloading
- Not quite correct algorithm for stopping automation
# 2022.6.3.0
*2022-06-03*
Changed version numbering method. From now on, new versions will be numbered by release date (YYYY.M.D)
**Attention! Starting with this release, SCrawler may not work on windows 7 and 8 or may not work correctly. All future releases will only be guaranteed to work on windows 10 and 11.**
- Added
- **Automation** (downloading data automatically every ```X``` minutes)
- Expanded settings for Instagram tagged posts that are downloaded for the first time.
- Fixed
- Videos hosted on Reddit that are downloaded via m3u8 playlists are missing an audio track.
- Instagram hash not able to be auto-filled from cookies
# 3.0.0.10
*2022-05-23*
- Added
- **Downloading groups**
- **Download saved Twitter posts** (bookmarks)
- Ability to enable/disable progress form opening at the start of downloading
- Ability to enable/disable Info form opening at the start of downloading
- The ability to disable the opening of forms Info and Progress at the start of downloads if it was once closed
- Focusing the main window when opening Info or Progress forms
- Ability to execute a script/command when closing SCrawler
- Ability to execute a script/command after all downloads are completed
- Minor improvements
- Fixed
- Instagram tagged data not downloading (now requires one more parameter **x-csrftoken** to download tagged data)
- In some cases, Instagram Stories cannot be downloaded due to forbidden Windows characters
- Separate Instagram posts were not downloading via the Video Downloader form.
- In some cases, an Imgur video hosted on Reddit won't download
- Gfycat data not downloading from saved Reddit posts
- In some cases, the date and time are not added to the filename
- Unable to download photos from Twitter in full resolution (4K)
# 3.0.0.9
*2022-04-24*
- Added
- Excluded labels
- Ability to disable user grouping
- Ability to show groups of user sites when filtering by labels
- Fixed
- Removed adding "No Parsed" internal label when not needed
- Redownloading Instagram Stories
# 3.0.0.8
*2022-04-19*
- Added
- Script mode ```command```
- Disabled Instagram error 403 (Forbidden) logging for downloading tagged data
- Fixed
- The script does not run after the user download is complete
# 3.0.0.7
*2022-04-14*
- Added
- Ability to run a script after the user download is complete
- Hotkey ```F2``` for additional options in the user creation form
- Fixed
- (Issue #32) In some cases, Date and Time are still not added for Stories and Tagged Photos
- (Issue #33) Instagram Stories downloading error
- LPSG downloader does not download all content
# 3.0.0.6
*2022-04-04*
- Added
- ```GoTo Start``` channels button
- ```GoTo End``` channels button
- Fixed
- In some cases, saved Reddit posts didn't fully download
- Incorrect Reddit accessibility check algorithm
- Incorrect behavior of the main progress bar when downloading saved posts
- (Issue #25) Date and Time not added for Stories and Tagged Photos
# 3.0.0.5
*2022-04-02*
- Added
- ```New```, ```Hot```, ```Top``` Reddit channel and user download modes
# 3.0.0.4
*2022-03-26*
- Fixed
- External plugins do not save information about downloaded files
- The user cannot be added to the collection if a special path has been specified.
# 3.0.0.3
*2022-03-24*
- Added
- Download all by specific sites
- Download all, ignoring the ```Ready for download``` option
- Download all by specific sites, ignoring the ```Ready for download``` option
- Fixed
- (Issue #19) Typo in default Instagram settings (Post limit timer)
- Typo when applying "Download UHD" in XVIDEOS plugin
- The sites filter does not work unless the "Fast profiles loading" option is enabled.
# 3.0.0.2
*2022-03-22*
- Added
- **LPSG** site plugin
- **XVIDEOS** site plugin
@@ -10,6 +563,8 @@
# 3.0.0.1
*2022-03-20*
- Added
- Download data up to a specific date
- Update and Reset functions in the plugin (ISiteSettings)
@@ -23,6 +578,8 @@
# 3.0.0.0
*2022-03-17*
**Attention! This version of the program makes changes user data file (Users.xml). Once you start using this version, you will not be able to use previous versions of the program. Therefore, it is highly recommended to archive the program settings folder and archive the users' data files (you can use the [```ArchiveSCrawlerUsersDataFiles.bat```](Tools/ArchiveSCrawlerUsersDataFiles.bat) tool to archive the data files of all users).**
- Added
@@ -63,6 +620,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 2.0.0.4
*2022-02-07*
**Removed compatibility of program settings with version 1.0.0.4 and lower.**
**If your program version is 1.0.0.4 and lower, it is strongly recommended that you upgrade to release 2.0.0.1 to update the program settings (and run the program). Then update to this release. Otherwise, you will have to configure the program settings again**
@@ -77,6 +636,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 2.0.0.3
*2022-02-02*
**Removed compatibility of program settings with version 1.0.0.4 and lower.**
**If your program version is 1.0.0.4 and lower, it is strongly recommended that you upgrade to release 2.0.0.1 to update the program settings (and run the program). Then update to this release. Otherwise, you will have to configure the program settings again**
@@ -92,6 +653,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 2.0.0.2
*2022-01-23*
**This is the last release that supports program settings of version 1.0.0.4 and lower. Compatibility of program settings with version 1.0.0.4 and lower will be removed in future releases. It is strongly recommended that you upgrade to this release before future releases. Otherwise, you will have to configure the program settings again. If your program version is 1.0.1.0 or higher, you should not pay attention to this message.**
- Added
@@ -114,6 +677,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 2.0.0.1
*2021-12-29*
- Added
- Download individual Imgur media files (use the "Download video" form).
- Fixed
@@ -122,6 +687,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 2.0.0.0
*2021-12-27*
- Added
- **Instagram**
- Filter by site
@@ -139,6 +706,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 1.0.1.0
*2021-12-20*
- Added
- Extended site settings
- Non-existend users will be marked in red
@@ -160,6 +729,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 1.0.0.4
*2021-12-12*
- Added
- Full channels support (you can now add channel (subreddit) for standard download)
- ```Ready for download``` now available for collections and can be changed for multiple user
@@ -168,12 +739,16 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 1.0.0.3
*2021-12-11*
- Fixed
- Custom "Download videos" option is not saved
- The "Download all" button is not activated after changing modes
# 1.0.0.2
*2021-12-10*
- Added
- Ability to choose what types of media you want to download (images only, videos only, both)
- Ability to name files by date
@@ -182,6 +757,8 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 1.0.0.1
*2021-12-09*
- Added
- Limited download if user added from the channel
- Forced limited download for any user
@@ -204,4 +781,6 @@ At the requests of some users, I added [screenshots](ProgramScreenshots) of the
# 1.0.0.0
*2021-12-07*
Initial release

71
FAQ.md
View File

@@ -2,6 +2,8 @@
**Please read the [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) Before asking a question!**
**Also read [here](README.md) for basic information.**
Most of your questions are already answered. All settings, functions, buttons and everything else described in the guide.
Any other questions I will keep in this file.
@@ -12,11 +14,7 @@ Any other questions I will keep in this file.
A: https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies
----
#### Q: **I can't copy cookies.**
A: Use the mouse. Don't use ```Ctrl``` + ```A```!
<!---**ATTENTION! If you need to use cookies but cannot import them, I highly recommend that you don't use SCrawler and use another program. Don't create issues, discussions, or write to me on Discord. Any issue or discussion about cookies will be deleted immediately without a response. Any user who asks me about cookies on Discord will be banned.**--->
----
@@ -28,19 +26,46 @@ A: This is a GUI program.
#### Q: **Will CLI be added in the future?**
A: I do not think so.
A: NO.
----
#### Q: **I want to add "...." site. How to request.**
A: How to request a new site you can read [here](CONTRIBUTING.md#how-to-request-a-new-site)
<!---A: How to request a new site you can read [here](CONTRIBUTING.md#how-to-request-a-new-site)--->
**I'm currently not accepting requests to develop new sites.**
----
#### Q: **Twitter/Instagram download failed.**
#### Q: **Site download failed.**
A: Check your credentials. Both of these sites require cookies. Check your [Twitter tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) and [Instagram settings](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-settings). If all settings are set, but nothing works, go to [create a new issue](https://github.com/AAndyProgram/SCrawler/issues). Don't forget to attach the LOG.
A: Check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set, but nothing works, go to [create a new issue](https://github.com/AAndyProgram/SCrawler/issues). Don't forget to attach the LOG.
**ATTENTION! Issues without URLs will be closed without a response!**
----
#### Q: **I have set credentials but still nothing is downloading**
A: Click the ```Start downloading``` button
----
#### Q: **Where can I find the release?**
A: https://github.com/AAndyProgram/SCrawler/releases/latest
----
#### Q: **How to run the program?**
A: Double-click ```SCrawler.exe```
----
#### Q: **Where to find binaries?**
A: https://github.com/AAndyProgram/SCrawler/releases/latest
----
@@ -52,7 +77,7 @@ A: The program stored posts IDs in users' folders. For the first time, the progr
#### Q: **How to redownload all data**
A: Double-click on the user you want to redownload. In the opened window open folder setting. Delete the files ending with ```_Data.xml``` and ```_Posts.txt```. Download this user again.
A: Double-click on the user you want to redownload. In the opened window open folder setting. Delete the files ending with ```_Data.xml``` and ```_Posts.txt```. Restart SCrawler. Download this user again.
----
@@ -64,4 +89,28 @@ A: There is no functionality to remove an individual label. You can open the ```
#### Q: **How to remove a user from the blacklist**
A: Just add that user back to the program. In the dialog box that opens, click on the ```Add and remove from blacklist```` button.
A: Just add that user back to the program. In the dialog box that opens, click on the ```Add and remove from blacklist``` button.
----
#### Q: **Why don't you answer how it works**
A: Because **I don't want to**. I don't want to waste my time explaining things that are already covered in the **[GUIDE](https://github.com/AAndyProgram/SCrawler/wiki)**! If you didn't bother to read the guide, why would I waste my time?! ALL FUNCTIONALITY IS DESCRIBED IN THE GUIDE. Before publishing a new release, I update the guide. If you don't respect my work, I don't waste my time.
----
#### Q: **You lost me. Your program is too complicated.**
A: **I'm fine with that**. If the program is difficult for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I am develop SCrawler for myself and publish on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm fine.
----
#### Q: **I can't configure something**
A: I can only [suggest](#q-you-lost-me-your-program-is-too-complicated) you find another (easier) program.
----
#### Q: **Can you add a step-by-step guide or video on how to use the program?**
A: **NO! NEVER!** The guide fully covers all the functionality of SCrawler! If you don't respect my work, I don't waste my time. If you want, you can create a video tutorial and send it to me. Then I add it. All options and what each option does described on the wiki. The wiki also contains a description of all settings and how-to configure them. For complex settings, there is a steep-by-steep guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I have developed a program with an intuitive interface. There is a Settings button, download buttons, a context menu that drops down when a user is clicked, and other controls. Anyone can use it.

View File

@@ -1,11 +1,12 @@
Your support is very valuable to me. Any support is greatly appreciated. Your support encourages me to make new features, update the program, add new sites, etc.
You can support the program by:
- **Bitcoin**: bitcoin:BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
- :heavy_dollar_sign: making donaion making donations on this site: https://ko-fi.com/andyprogram
- **Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
- :repeat: make a post about my program on your profile (Reddit, Twitter, Instagram and any other social networks)
- :speech_balloon: tell your friends about the program
- :heart: like the program on this site: https://alternativeto.net/software/scrawler/about/
- :heart: like the program on this site: https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml
- suggest my program as an alternative ([on this site](https://alternativeto.net/software/scrawler/about/)) to any program you have used before
I would be very grateful for any support! :blush:
I would be very grateful for any support! :blush:
<!---:heavy_dollar_sign: make a donation on this site: https://ko-fi.com/andyprogram--->

View File

@@ -1,3 +1,4 @@
List of available plugins:
- LPSG
- XVIDEOS
Tools:
- [image2post](https://github.com/unknown81311/SCrawler-image2post) by @unknown81311: **get reddit post URL from file.**

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 483 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 11 KiB

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 369 KiB

After

Width:  |  Height:  |  Size: 363 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 377 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.5 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.3 KiB

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.9 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.0 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

View File

@@ -1,10 +1,61 @@
# yt-dlp
https://github.com/yt-dlp/yt-dlp/
**Great powerful CLI tool that supports hundreds of sites.**
SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.
# 4K Video Downloader
https://www.4kdownload.com/-plbrz/video-downloader
| Option | SCrawler | 4K Stogram |
| ---- | ---- | ---- |
| User managament | **Advanced** | No |
| Automatic downloads | **Yes** | No |
| Downloading groups | **Yes** | No |
| Labeling users | **Yes** | No |
| Filtering | **Yes** | No |
| Collections | **Yes** | No |
| Specific user folders | **Yes** | No |
| Favorite / Temporary user options | **Yes** | No |
| Plugins support | **Yes** | No |
| Download single video | **Unlimited** | 30 videos per day *(unlimited starts from 12 EUR)* |
| Download videos per channel | **Unlimited** | 5 free *(unlimited starts from 12 EUR)* |
| Download videos per playlist | **Unlimited** | 10 free *(unlimited starts from 12 EUR)* |
| Download video subtitles | **Any for free**: single video, playlist, user/channel, album, etc| Free for single video |
| The number of subtitles you can download for a video | **All of them** | Up to 10 |
| Convert subtitles to additional formats | **Yes** | No |
| Support LRC format | **Yes** | No |
| Select audio codec for audio/video | **Yes** | No |
| Extract and convert additional audio tracks for video | **Yes** | No |
| Simultaneous downloads | **Unlimited** | 1 free, 3 for 12 EUR, 7 for 43 EUR|
| Private YouTube content download | **Free** | Only in paid plans *starts from 12 EUR* |
| **Paid** | **No** | Yes |
| **Free options** | **The program is completely free** | Only **30** videos per day, 5 from a channel, 10 from a playlist |
| Permitted Commercial Use | **Yes** | Starting from 43 EUR |
| Automatic Subscriptions Update | **Free** | Paid (43 EUR) |
| Posts and Captions Export | No | Paid (43 EUR) |
| Advertisements free | **No ADs at all for free** | Paid (43 EUR) |
| Operating Systems | Windows 10+ | Windows 7+, MacOS 10.13+, Ubuntu x64 |
| Select want content type to download | **Yes** | No |
| Instagram support | **Yes** | No |
| Twitter support | **Yes** | No |
| Reddit support | **Yes** | No |
| Other sites support | **Yes** | No |
| Still supported | Yes | Yes |
# 4K Stogram
https://www.4kdownload.com/products/product-stogram
<!---https://www.4kdownload.com/products/product-stogram--->
https://www.4kdownload.com/-ad0p9/stogram
| Option | SCrawler | 4K Stogram |
| ---- | ---- | ---- |
| User managament | **Advanced** | Primitive |
| Automatic downloads | **Yes** | No |
| Downloading groups | **Yes** | No |
| Labeling users | **Yes** | No |
| Filtering | **Yes** | No |
| Collections | **Yes** | No |
@@ -16,20 +67,20 @@ https://www.4kdownload.com/products/product-stogram
| Download posts by location | No | **Yes** |
| Save Private Instagram Content with Permission| Yes | Yes |
| Download Instagram Stories and Highlights | Yes | Yes |
| See Others Instagram Feed As Your Own | No | **Yes** |
| See Others Instagram Feed As Your Own | Yes | Yes |
| Download Instagram Video Posts | Yes | Yes |
| Backup Your Instagram Account | Yes | Yes |
| Save Instagram Posts by Date | No (only limited download) | **Yes** |
| Save Instagram Posts by Date | Yes | Yes |
| Download Instagram Saved Posts | Yes | Yes |
| Download Instagram Tagged Posts | Yes | Yes |
| Export and import subscriptions | No | **Yes** |
| **Paid** | **No** | Yes |
| **Free options** | **The program is completely free** | Only **ONE** profile downloading and only **200 posts** per day |
| Permitted Commercial Use | **Yes** | Starting from 43.56 EUR |
| Automatic Subscriptions Update | **Free** | Paid (43.56 EUR) |
| Posts and Captions Export | No | Paid (43.56 EUR) |
| Advertisements free | **No ADs at all for free** | Paid (14.52) |
| Operating Systems | Windows 7+ | Windows 7+, MacOS 10.13+, Ubuntu x64 |
| Permitted Commercial Use | **Yes** | Starting from 43 EUR |
| Automatic Subscriptions Update | **Free** | Paid (43 EUR) |
| Posts and Captions Export | No | Paid (43 EUR) |
| Advertisements free | **No ADs at all for free** | Paid (18 EUR) |
| Operating Systems | Windows 10+ | Windows 7+, MacOS 10.13+, Ubuntu x64 |
| Select want content type to download | **Yes** | No |
| Instagram support | Yes | Yes |
| Twitter support | **Yes** | No |
@@ -44,6 +95,8 @@ https://github.com/RipMeApp/ripme
| Option | SCrawler | RipMeApp |
| ---- | ---- | ---- |
| User managament | **Advanced** | No |
| Automatic downloads | **Yes** | No |
| Downloading groups | **Yes** | No |
| Labeling users | **Yes** | No |
| Filtering | **Yes** | No |
| Collections | **Yes** | No |
@@ -62,9 +115,9 @@ https://github.com/RipMeApp/ripme
| Export and import subscriptions | No | No |
| **Paid** | **No** | **No** |
| **Free options** | The program is completely free | The program is completely free, but site limits are not declared |
| Operating Systems | Windows 7+ | Windows, MacOS, Linux |
| Operating Systems | Windows 10+ | Windows, MacOS, Linux |
| Select want content type to download | Yes | Yes |
| Suported sites | 3 internal and any site using plugins | 86+ sites (declared) |
| Suported sites | 15 internal and any site using plugins | 86+ sites (declared) |
| Other sites support | **Yes** | No |
| Still supported | **Yes** | **No (last release date May 4, 2021)** |
@@ -72,7 +125,6 @@ https://github.com/RipMeApp/ripme
https://github.com/mikf/gallery-dl
**CLI tool**
**CLI tool**! Configured with JSON files only. Users need to learn complex configuration options, JSON, commands to use that tool. Very difficult to configure.
SCrawler has advanced user management, collections, labels, groups, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.
SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.

197
README.md
View File

@@ -1,92 +1,144 @@
# Social networks crawler
# :rainbow_flag: Happy LGBT Pride Month :tada:
# :rainbow_flag: Social networks crawler :rainbow_flag:
[![GitHub release (latest by date)](https://img.shields.io/github/v/release/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/releases/latest)
[![GitHub](https://img.shields.io/github/license/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/)
[![GitHub license](https://img.shields.io/github/license/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/blob/main/LICENSE)
[![GitHub all releases](https://img.shields.io/github/downloads/aandyprogram/scrawler/total?label=Total%20downloads)](https://github.com/AAndyProgram/SCrawler/releases)
[![FAQ](https://img.shields.io/badge/FAQ-green)](FAQ.md)
[![GUIDE](https://img.shields.io/badge/GUIDE-green)](https://github.com/AAndyProgram/SCrawler/wiki)
[![How to support](https://img.shields.io/badge/HowToSupport-green)](HowToSupport.md)
:eu:
:greece:
A program to download photo and video from [any site](#supported-sites) (e.g. Reddit, Twitter, Instagram).
A program to download photo and video from [any site](#supported-sites) (e.g. YouTube, YouTube Music, Reddit, Twitter, Mastodon, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, ThisVid, LPSG, Pinterest).
Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :blush:
[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)
**Bitcoin**: bitcoin:BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
**If you like SCrawler, please like the program on [this site](https://alternativeto.net/software/scrawler/about/) and/or [this](https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml)**
<!---Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :blush:
[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)--->
**Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
![Main window](ProgramScreenshots/MainWindow.png)
![Channels window](ProgramScreenshots/Channels.png)
[**YouTube standalone application:**](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader)
![YouTube application](ProgramScreenshots/AppYouTube.png)
# What can program do:
- Download pictures and videos from users' profiles and subreddits:
- Reddit images;
- Reddit galleries of images;
- Redgifs hosted videos (https://www.redgifs.com/);
- Reddit hosted videos (downloading Reddit hosted video is going through ffmpeg (**ffmpeg only works with the x64 program**));
- Twitter images and videos;
- Instagram images and videos.
- Imgur images, galleries and videos
- Gfycat videos
- YouTube videos, shorts, users, artists, playlists, music, tracks;
- Reddit images, galleries of images, videos, saved posts;
- Redgifs videos (https://www.redgifs.com/);
- Twitter images and videos, saved (bookmarked) posts;
- Mastodon images and videos, saved (bookmarked) posts;
- Instagram images and videos, tagged posts, stories, saved posts;
- TikTok videos (*currently broken*; [limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits));
- Pinterest boards, users, saved posts;
- Imgur images, galleries and videos;
- Gfycat videos;
- PornHub images, videos, save (liked) posts;
- XHamster images, videos, saved posts;
- XVIDEOS videos, saved posts;
- ThisVid images, videos, saved posts;
- [Other](#supported-sites) supported sites
- Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels).
- Download [saved Reddit and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts).
- Add users from parsed channel.
- **Advanced user management.**
- Labeling users.
- Adding users to favorites and temporary.
- Filter exists users by label or group.
- Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
- Download [saved Reddit, Twitter and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
- Add users from parsed channel
- **Advanced user management**
- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every ```X``` minutes)
- **Feed** ([feed](https://github.com/AAndyProgram/SCrawler/wiki#feed) of downloaded media files)
- Labeling users
- Create [download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
- Adding users to favorites and temporary
- [Filter exists users](https://github.com/AAndyProgram/SCrawler/wiki#view) by label or group
- Selection of media types you want to download (images only, videos only, both)
- Download a special video, image or gallery
- Making collections (grouping users into collections)
- [Download a special video](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video), image or gallery
- Making [collections](https://github.com/AAndyProgram/SCrawler/wiki#collections) (grouping users into collections)
- Specifying a user folder (for downloading data to another location)
- Changing user icons
- Changing view modes
- Changing [view modes](https://github.com/AAndyProgram/SCrawler/wiki#view)
- ...and many others...
# Supported sites
- **YouTube**
- **YouTube Music**
- **Reddit**
- **Twitter**
- **Mastodon**
- **Instagram**
- TikTok (*currently broken*; [limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
- RedGifs
- Pinterest
- Imgur
- Gfycat
- LPSG
- XVIDEOS
- **PornHub**
- **XHamster**
- **XVIDEOS**
- **ThisVid**
- [Other sites](Plugins.md)
# How does it works:
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
# How it works
First, the program downloads the full profile. After the program downloads only new posts. The program remembers downloaded posts.
## Reddit
The program parses all user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
The program parses user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
## Other sites
The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
You can read about Instagram restrictions [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-limits)
The program parses user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
## How to request a new site
Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
<!---Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about--->
**I'm currently not accepting requests to develop new sites.**
# Requirements
- Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher
- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) for Twitter (if you want to download data from Twitter)
- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [Hash](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) for Instagram (if you want to download data from Instagram), [Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash-2) for saved Instagram posts, Instagram [stories authorization headers](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-stories-authorization-headers) for Stories and Tagged data
- ffmpeg library for downloading videos hosted on Reddit (you can download it from the [official repo](https://github.com/GyanD/codexffmpeg/releases/tag/2021-01-12-git-ca21cb1e36) or [from my first release](https://github.com/AAndyProgram/SCrawler/releases/download/1.0.0.0/ffmpeg.zip)). **ffmpeg only works with the x64 version of the program.**
- Windows 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
# Guide
- [Main window](https://github.com/AAndyProgram/SCrawler/wiki)
- [Users](https://github.com/AAndyProgram/SCrawler/wiki/Users)
- [Add/Edit/Delete users](https://github.com/AAndyProgram/SCrawler/wiki/Users)
- [Collections](https://github.com/AAndyProgram/SCrawler/wiki#collections)
- [User operations](https://github.com/AAndyProgram/SCrawler/wiki#context-menu)
- [User labels](https://github.com/AAndyProgram/SCrawler/wiki/Users#labels)
- **[DOWNLOAD](https://github.com/AAndyProgram/SCrawler/wiki#download)**
- [Automation](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation)
- [Download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
- [Downloading information](https://github.com/AAndyProgram/SCrawler/wiki#info)
- [Reddit channels](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
- [Saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)
- [View modes, filters](https://github.com/AAndyProgram/SCrawler/wiki#view)
- **[SETTINGS](https://github.com/AAndyProgram/SCrawler/wiki/Settings)**
- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
- [Reddit](https://github.com/AAndyProgram/SCrawler/wiki/Settings#reddit)
- [Twitter](https://github.com/AAndyProgram/SCrawler/wiki/Settings#twitter)
- [Mastodon](https://github.com/AAndyProgram/SCrawler/wiki/Settings#Mastodon)
- [Instagram](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram)
- [TikTok](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok)
- [RedGifs](https://github.com/AAndyProgram/SCrawler/wiki/Settings#redgifs)
- [YouTube](https://github.com/AAndyProgram/SCrawler/wiki/Settings#YouTube)
- [Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#Pinterest)
- [PornHub](https://github.com/AAndyProgram/SCrawler/wiki/Settings#pornhub)
- [XHamster](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xhamster)
- [XVIDEOS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xvideos)
- [ThisVid](https://github.com/AAndyProgram/SCrawler/wiki/Settings#ThisVid)
- [LPSG](https://github.com/AAndyProgram/SCrawler/wiki/Settings#lpsg)
**Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
# Installation
**Just unzip the program archive to any folder, copy the file ```ffmpeg.exe``` into it and enjoy.** :blush:
**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder and enjoy.** :blush:
**Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
@@ -94,53 +146,48 @@ Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about
Just download [latest](https://github.com/AAndyProgram/SCrawler/releases/latest) version and unpack it into the program folder. **Before starting a new version, I recommend making a backup copy of the program settings folder.**
# How to build from source
# [How to build from source](CONTRIBUTING.md#how-to-build-from-source)
1. Delete the "PersonalUtilities" project from the solution.
1. Add the latest version of the "PersonalUtilities.dll" library (from the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest)).
1. Import PersonalUtilities.Functions for the whole project.
# [How to make a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins)
# How to make a plugin
Read about how to make plugin [here](https://github.com/AAndyProgram/SCrawler/wiki/Plugins).
# How to support
Read more about how to support the program [here](HowToSupport.md).
# [How to support](HowToSupport.md)
# Settings and usage
The program has an intuitive interface.
You need to set up authorization for Twitter and Instagram:
- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [tokens](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-twitter-tokens) for **Twitter** (if you want to download data from Twitter)
- Authorization [cookies](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies) and [Hash](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram) for **Instagram** (if you want to download data from Instagram), [Hash 2](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-hash-2) for **saved Instagram posts**, Instagram [stories authorization headers](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-find-instagram-stories-authorization-headers) for **Stories** and **Tagged data**
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
Just add a user profile and click the ```Start downloading``` button.
Just add a user profile and **click the ```Download``` button**.
You can add users by patterns:
- https://www.instagram.com/SomeUserName
- https://twitter.com/SomeUserName
- https://reddit.com/user/SomeUserName
- https://reddit.com/r/SomeSubredditName
- https://www.redgifs.com/users/SomeUserName
- u/SomeUserName
- r/SomeSubredditName
- SomeUserName (in this case, you need to select the user's site)
- SomeSubredditName
Read more about adding users and subreddits [here](https://github.com/AAndyProgram/SCrawler/wiki/Users)
```mermaid
stateDiagram
Start: Add site credentials
What: What would I like to do
DownUser: Download user
DownVideo: Download video
AUser: Add user (1)
OVIF: Open standalone downloader (2)
AVideo: Add video url
F5: Press 'F5' or click the download button
[*]-->Start
Start-->What
What-->DownUser
What-->DownVideo
DownUser-->AUser
DownVideo-->OVIF
OVIF-->AVideo
AVideo-->F5
AUser-->F5
F5-->[*]
```
1. Press `Insert` or click the `Download` button ([read more here](https://github.com/AAndyProgram/SCrawler/wiki#users-list), [hot keys](https://github.com/AAndyProgram/SCrawler/wiki#hot-keys))
2. Click the `Download` button, then `Standalone downloader` ([read more here](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video))
![Add user](ProgramScreenshots/CreateUserClear.png)
# Using program as just video downloader
Create a shortcut for the program. Open shortcut properties. In the ```Shortcut``` tab, in the ```Target``` field, just add the letter ```v``` at the end across the space.
Example: ```D:\Programs\SCrawler\SCrawler.exe v```
![Separate video downloader](ProgramScreenshots/SeparateVideoDownloader.png)
# Contact me
[![matrix](https://img.shields.io/badge/Matrix-%40andyprogram%3Amatrix.org-informational)](https://matrix.to/#/@andyprogram:matrix.org)
Matrix (Element): https://matrix.to/#/@andyprogram:matrix.org
Discord: AndyProgram#3804

View File

@@ -9,10 +9,15 @@
Imports PersonalUtilities.Functions.RegularExpressions
Friend Module Declarations
Friend ReadOnly Property PhotoRegEx As RParams = RParams.DM("(https://www.lpsg.com/attachments)(.+?)(?="")", 0, RegexReturn.List)
Friend ReadOnly Property PhotoRegExExt As New RParams("img.data.src=""(/proxy[^""]+?)""", Nothing, 1, RegexReturn.List) With {
.Converter = Function(Input) $"https://www.lpsg.com/{SymbolsConverter.HTML.Decode(Input)}"}
Friend ReadOnly Property NextPageRegex As RParams = RParams.DMS("<link rel=""next"" href=""(.+?/page-(\d+))""", 2)
Private Const FileUrlRegexDefault As String = "(.+[^/]+?)(jpg|jpeg|gif|png)"
Private Const FileUrlRegexDefault As String = "([^/]+?)(jpg|jpeg|gif|png|webm)"
Private ReadOnly InputFReplacer As New ErrorsDescriber(EDP.ReturnValue)
Private ReadOnly InputForbidRemover As Func(Of String, String) = Function(Input) If(Input.IsEmptyString, Input, Input.StringRemoveWinForbiddenSymbols(, InputFReplacer))
Friend ReadOnly Property FileRegEx As New RParams(FileUrlRegexDefault, Nothing, 0) With {
.Converter = Function(ByVal Input As String) As String
Input = InputForbidRemover.Invoke(Input)
If Not Input.IsEmptyString Then
Dim lv$ = Input.Split("-").LastOrDefault
If Not lv.IsEmptyString Then
@@ -22,5 +27,8 @@ Friend Module Declarations
End If
Return Input
End Function}
Friend ReadOnly Property FileRegExExt As New RParams(FileUrlRegexDefault, 0, Nothing, InputForbidRemover)
Friend ReadOnly Property FileRegExExt2 As New RParams("([^/]+?)(?=(\Z|&))", 0, Nothing, InputForbidRemover)
Friend ReadOnly Property FileExistsRegEx As RParams = RParams.DMS(FileUrlRegexDefault, 2)
Friend ReadOnly TempListAddParams As New ListAddParams(LAP.NotContainsOnly) With {.Comparer = New FComparer(Of PluginUserMedia)(Function(x, y) x.URL = y.URL)}
End Module

View File

@@ -13,7 +13,7 @@ Imports System.Runtime.InteropServices
<Assembly: AssemblyDescription("LPSG plugin for SCrawler")>
<Assembly: AssemblyCompany("AndyProgram")>
<Assembly: AssemblyProduct("LPSG")>
<Assembly: AssemblyCopyright("Copyright © 2022")>
<Assembly: AssemblyCopyright("Copyright © 2022")>
<Assembly: AssemblyTrademark("AndyProgram")>
<Assembly: ComVisible(False)>
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("1.0.0.0")>
<Assembly: AssemblyFileVersion("1.0.0.0")>
<Assembly: AssemblyVersion("2022.10.12.0")>
<Assembly: AssemblyFileVersion("2022.10.12.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -30,7 +30,7 @@ Public Class SiteSettings : Implements ISiteSettings
.LoadSettings()
Else
.CookiesDomain = "www.lpsg.com"
.Cookies = New CookieKeeper("www.lpsg.com")
.Cookies = New CookieKeeper(.CookiesDomain)
End If
End With
End Sub
@@ -54,6 +54,10 @@ Public Class SiteSettings : Implements ISiteSettings
End Sub
#End Region
#Region "Update"
Public Sub BeginEdit() Implements ISiteSettings.BeginEdit
End Sub
Public Sub EndEdit() Implements ISiteSettings.EndEdit
End Sub
Public Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
End Sub
Public Sub EndUpdate() Implements ISiteSettings.EndUpdate
@@ -81,7 +85,7 @@ Public Class SiteSettings : Implements ISiteSettings
Else
Return Nothing
End If
Catch ex As Exception
Catch
Return Nothing
End Try
End Function
@@ -91,10 +95,13 @@ Public Class SiteSettings : Implements ISiteSettings
Public Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable(Of PluginUserMedia) Implements ISiteSettings.GetSpecialData
Return Nothing
End Function
Public Function Available(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.Available
Return True
Public Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
Return If(Responser.Cookies?.Count, 0) > 0
End Function
Public Function ReadyToDownload(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.ReadyToDownload
Return True
End Function
Public Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String Implements ISiteSettings.GetUserPostUrl
Return String.Empty
End Function
End Class

View File

@@ -7,6 +7,9 @@
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Imports PersonalUtilities.Functions.RegularExpressions
Imports UStates = SCrawler.Plugin.PluginUserMedia.States
Imports UTypes = SCrawler.Plugin.PluginUserMedia.Types
Imports Converters = PersonalUtilities.Functions.SymbolsConverter.Converters
Public Class UserData : Implements IPluginContentProvider
#Region "XML names"
Private Const Name_LatestPage As String = "LatestPage"
@@ -35,7 +38,8 @@ Public Class UserData : Implements IPluginContentProvider
Public Property SeparateVideoFolder As Boolean Implements IPluginContentProvider.SeparateVideoFolder
Public Property DataPath As String Implements IPluginContentProvider.DataPath
Public Property PostsNumberLimit As Integer? Implements IPluginContentProvider.PostsNumberLimit
Public Property PostsDateLimit As Date? Implements IPluginContentProvider.PostsDateLimit
Public Property DownloadDateFrom As Date? Implements IPluginContentProvider.DownloadDateFrom
Public Property DownloadDateTo As Date? Implements IPluginContentProvider.DownloadDateTo
#End Region
#Region "Interface exchange options"
Public Sub ExchangeOptionsSet(ByVal Obj As Object) Implements IPluginContentProvider.ExchangeOptionsSet
@@ -58,14 +62,14 @@ Public Class UserData : Implements IPluginContentProvider
#End Region
Private Property LatestPage As String = String.Empty
Private Property Responser As Response = Nothing
Private Enum Mode : Internal : External : End Enum
Public Sub GetMedia() Implements IPluginContentProvider.GetMedia
Try
If Not Responser Is Nothing Then Responser.Dispose()
Responser = New Response
With Responser : .Copy(Settings.Responser) : .Error = EDP.ThrowException : End With
Dim l As List(Of String) = Nothing
Dim NextPage$ = String.Empty
Dim NextPage$
Dim r$
Dim _LPage As Func(Of String) = Function() If(LatestPage.IsEmptyString, String.Empty, $"page-{LatestPage}")
@@ -76,35 +80,60 @@ Public Class UserData : Implements IPluginContentProvider
Thrower.ThrowAny()
If Not r.IsEmptyString Then
NextPage = RegexReplace(r, NextPageRegex)
l.ListAddList(RegexReplace(r, PhotoRegEx), LAP.NotContainsOnly)
UpdateMediaList(RegexReplace(r, PhotoRegEx), Mode.Internal)
UpdateMediaList(RegexReplace(r, PhotoRegExExt), Mode.External)
If NextPage = LatestPage Or NextPage.IsEmptyString Then Exit Do Else LatestPage = NextPage
Else
Exit Do
End If
Loop
If l.ListExists Then
Dim f As SFile
For Each u$ In l
If Not IsEmptyString(RegexReplace(u, FileExistsRegEx)) Then
f = CStr(RegexReplace(u, FileRegEx))
f.Path = DataPath.CSFilePSN
f.Separator = "\"
TempMediaList.Add(New PluginUserMedia With {.ContentType = PluginUserMedia.Types.Picture, .URL = u, .File = f})
End If
Next
If TempMediaList.ListExists And ExistingContentList.ListExists Then _
TempMediaList.RemoveAll(Function(m) ExistingContentList.Exists(Function(mm) mm.URL = m.URL))
End If
If TempMediaList.ListExists And ExistingContentList.ListExists Then _
TempMediaList.RemoveAll(Function(m) ExistingContentList.Exists(Function(mm) mm.URL = m.URL))
Catch oex As OperationCanceledException
Catch dex As ObjectDisposedException
Catch ex As Exception
LogProvider.Add(ex, "[LPSG.UserData.GetMedia]")
If Responser.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
LogProvider.Add("LPSG not available")
Else
LogProvider.Add(ex, $"[LPSG.UserData.GetMedia({Name})]")
End If
End Try
End Sub
Private Sub UpdateMediaList(ByVal l As List(Of String), ByVal m As Mode)
If l.ListExists Then
Dim f As SFile
Dim u$
Dim exists As Boolean
Dim r As RParams
Dim ude As New ErrorsDescriber(EDP.ReturnValue)
For Each url$ In l
If Not url.IsEmptyString Then u = SymbolsConverter.Decode(url, {Converters.HTML, Converters.ASCII}, ude) Else u = String.Empty
If Not u.IsEmptyString Then
exists = Not IsEmptyString(RegexReplace(u, FileExistsRegEx))
If m = Mode.Internal Then
r = FileRegEx
Else
r = FileRegExExt
If Not exists Then
r = FileRegExExt2
exists = Not IsEmptyString(RegexReplace(u, FileRegExExt2))
End If
End If
If exists Then
f = CStr(RegexReplace(u, r))
f.Path = DataPath.CSFilePSN
f.Separator = "\"
If f.Extension.IsEmptyString Then f.Extension = "jpg"
TempMediaList.ListAddValue(New PluginUserMedia With {.ContentType = UTypes.Picture, .URL = url, .File = f}, TempListAddParams)
End If
End If
Next
End If
End Sub
Public Sub Download() Implements IPluginContentProvider.Download
Try
With Responser : .UseWebClient = True : .UseWebClientCookies = True : End With
With Responser : .UseWebClient = True : .UseWebClientCookies = True : .ResetError() : End With
If TempMediaList.ListExists Then
Dim m As PluginUserMedia
Dim eweb As ErrorsDescriber = EDP.ThrowException
@@ -112,15 +141,21 @@ Public Class UserData : Implements IPluginContentProvider
For i% = 0 To TempMediaList.Count - 1
Thrower.ThrowAny()
m = TempMediaList(i)
m.DownloadState = PluginUserMedia.States.Tried
m.DownloadState = UStates.Tried
Try
If Not m.URL.IsEmptyString And Not m.File.IsEmptyString Then
Responser.DownloadFile(m.URL, m.File, eweb)
m.DownloadState = PluginUserMedia.States.Downloaded
m.DownloadState = UStates.Downloaded
Else
m.DownloadState = PluginUserMedia.States.Skipped
m.DownloadState = UStates.Skipped
End If
Catch wex As Exception
If Responser.Client.StatusCode = Net.HttpStatusCode.ServiceUnavailable Then
LogProvider.Add("LPSG not available")
Else
m.DownloadState = UStates.Missing
m.Attempts += 1
End If
Catch ex As Exception
End Try
RaiseEvent ProgressChanged(1)
TempMediaList(i) = m

View File

@@ -41,13 +41,13 @@ Friend NotInheritable Class M3U8
Return Nothing
Catch ex As Exception
Logger.Add(ex, "[M3U8.Save]")
Return Nothing
ex.HelpLink = 1
Throw ex
Finally
CachePath.Delete(SFO.Path, SFODelete.None, EDP.None)
End Try
End Function
Friend Shared Function Download(ByVal URL As String, ByVal Appender As String, ByVal ffmpegFile As SFile, ByVal f As SFile,
ByRef Logger As ILogProvider) As SFile
Friend Shared Function Download(ByVal URL As String, ByVal Appender As String, ByVal ffmpegFile As SFile, ByVal f As SFile, ByRef Logger As ILogProvider) As SFile
Try
If Not URL.IsEmptyString Then
Using w As New WebClient
@@ -61,8 +61,8 @@ Friend NotInheritable Class M3U8
End If
Return Nothing
Catch ex As Exception
Logger.Add(ex, "[M3U8.Download]")
Return Nothing
If Not ex.HelpLink = 1 Then Logger.Add(ex, $"[M3U8.Download({URL}, {Appender}, {ffmpegFile}, {f})]")
Throw ex
End Try
End Function
End Class

View File

@@ -13,7 +13,7 @@ Imports System.Runtime.InteropServices
<Assembly: AssemblyDescription("XVIDEOS plugin for SCrawler")>
<Assembly: AssemblyCompany("AndyProgram")>
<Assembly: AssemblyProduct("XVIDEOS")>
<Assembly: AssemblyCopyright("Copyright © 2022")>
<Assembly: AssemblyCopyright("Copyright © 2022")>
<Assembly: AssemblyTrademark("AndyProgram")>
<Assembly: ComVisible(False)>
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("1.0.0.0")>
<Assembly: AssemblyFileVersion("1.0.0.0")>
<Assembly: AssemblyVersion("2022.10.12.0")>
<Assembly: AssemblyFileVersion("2022.10.12.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -25,14 +25,9 @@ Partial Public Class SettingsForm : Inherits System.Windows.Forms.Form
Dim CONTAINER_MAIN As System.Windows.Forms.ToolStripContainer
Dim resources As System.ComponentModel.ComponentResourceManager = New System.ComponentModel.ComponentResourceManager(GetType(SettingsForm))
Me.LIST_DOMAINS = New System.Windows.Forms.ListBox()
Me.ToolbarTOP = New System.Windows.Forms.ToolStrip()
Me.BTT_ADD = New System.Windows.Forms.ToolStripButton()
Me.BTT_DELETE = New System.Windows.Forms.ToolStripButton()
CONTAINER_MAIN = New System.Windows.Forms.ToolStripContainer()
CONTAINER_MAIN.ContentPanel.SuspendLayout()
CONTAINER_MAIN.TopToolStripPanel.SuspendLayout()
CONTAINER_MAIN.SuspendLayout()
Me.ToolbarTOP.SuspendLayout()
Me.SuspendLayout()
'
'CONTAINER_MAIN
@@ -41,7 +36,7 @@ Partial Public Class SettingsForm : Inherits System.Windows.Forms.Form
'CONTAINER_MAIN.ContentPanel
'
CONTAINER_MAIN.ContentPanel.Controls.Add(Me.LIST_DOMAINS)
CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(384, 266)
CONTAINER_MAIN.ContentPanel.Size = New System.Drawing.Size(384, 291)
CONTAINER_MAIN.Dock = System.Windows.Forms.DockStyle.Fill
CONTAINER_MAIN.LeftToolStripPanelVisible = False
CONTAINER_MAIN.Location = New System.Drawing.Point(0, 0)
@@ -50,52 +45,15 @@ Partial Public Class SettingsForm : Inherits System.Windows.Forms.Form
CONTAINER_MAIN.Size = New System.Drawing.Size(384, 291)
CONTAINER_MAIN.TabIndex = 0
'
'CONTAINER_MAIN.TopToolStripPanel
'
CONTAINER_MAIN.TopToolStripPanel.Controls.Add(Me.ToolbarTOP)
'
'LIST_DOMAINS
'
Me.LIST_DOMAINS.Dock = System.Windows.Forms.DockStyle.Fill
Me.LIST_DOMAINS.FormattingEnabled = True
Me.LIST_DOMAINS.Location = New System.Drawing.Point(0, 0)
Me.LIST_DOMAINS.Name = "LIST_DOMAINS"
Me.LIST_DOMAINS.Size = New System.Drawing.Size(384, 266)
Me.LIST_DOMAINS.Size = New System.Drawing.Size(384, 291)
Me.LIST_DOMAINS.TabIndex = 0
'
'ToolbarTOP
'
Me.ToolbarTOP.Dock = System.Windows.Forms.DockStyle.None
Me.ToolbarTOP.GripStyle = System.Windows.Forms.ToolStripGripStyle.Hidden
Me.ToolbarTOP.Items.AddRange(New System.Windows.Forms.ToolStripItem() {Me.BTT_ADD, Me.BTT_DELETE})
Me.ToolbarTOP.Location = New System.Drawing.Point(0, 0)
Me.ToolbarTOP.Name = "ToolbarTOP"
Me.ToolbarTOP.Size = New System.Drawing.Size(384, 25)
Me.ToolbarTOP.Stretch = True
Me.ToolbarTOP.TabIndex = 0
'
'BTT_ADD
'
Me.BTT_ADD.AutoToolTip = False
Me.BTT_ADD.BackColor = System.Drawing.Color.FromArgb(CType(CType(192, Byte), Integer), CType(CType(255, Byte), Integer), CType(CType(192, Byte), Integer))
Me.BTT_ADD.ForeColor = System.Drawing.Color.DarkGreen
Me.BTT_ADD.Image = CType(resources.GetObject("BTT_ADD.Image"), System.Drawing.Image)
Me.BTT_ADD.ImageTransparentColor = System.Drawing.Color.Magenta
Me.BTT_ADD.Name = "BTT_ADD"
Me.BTT_ADD.Size = New System.Drawing.Size(49, 22)
Me.BTT_ADD.Text = "Add"
'
'BTT_DELETE
'
Me.BTT_DELETE.AutoToolTip = False
Me.BTT_DELETE.BackColor = System.Drawing.Color.FromArgb(CType(CType(255, Byte), Integer), CType(CType(192, Byte), Integer), CType(CType(192, Byte), Integer))
Me.BTT_DELETE.ForeColor = System.Drawing.Color.Maroon
Me.BTT_DELETE.Image = CType(resources.GetObject("BTT_DELETE.Image"), System.Drawing.Image)
Me.BTT_DELETE.ImageTransparentColor = System.Drawing.Color.Magenta
Me.BTT_DELETE.Name = "BTT_DELETE"
Me.BTT_DELETE.Size = New System.Drawing.Size(60, 22)
Me.BTT_DELETE.Text = "Delete"
'
'SettingsForm
'
Me.AutoScaleDimensions = New System.Drawing.SizeF(6.0!, 13.0!)
@@ -112,18 +70,11 @@ Partial Public Class SettingsForm : Inherits System.Windows.Forms.Form
Me.SizeGripStyle = System.Windows.Forms.SizeGripStyle.Hide
Me.Text = "Settings"
CONTAINER_MAIN.ContentPanel.ResumeLayout(False)
CONTAINER_MAIN.TopToolStripPanel.ResumeLayout(False)
CONTAINER_MAIN.TopToolStripPanel.PerformLayout()
CONTAINER_MAIN.ResumeLayout(False)
CONTAINER_MAIN.PerformLayout()
Me.ToolbarTOP.ResumeLayout(False)
Me.ToolbarTOP.PerformLayout()
Me.ResumeLayout(False)
End Sub
Private WithEvents LIST_DOMAINS As Windows.Forms.ListBox
Private WithEvents ToolbarTOP As Windows.Forms.ToolStrip
Private WithEvents BTT_ADD As Windows.Forms.ToolStripButton
Private WithEvents BTT_DELETE As Windows.Forms.ToolStripButton
End Class

View File

@@ -120,60 +120,7 @@
<metadata name="CONTAINER_MAIN.GenerateMember" type="System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
<value>False</value>
</metadata>
<metadata name="ToolbarTOP.TrayLocation" type="System.Drawing.Point, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<value>17, 17</value>
</metadata>
<assembly alias="System.Drawing" name="System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
<data name="BTT_ADD.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
<value>
iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8
YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAN+SURBVEhLrZVZSFRRGMdvKa6oE5Y1bpTtRY2pJYk3ayyd
xgrvaDXlQotLKVqSo7QR2QIREUX0EgUSET1V9NTyYBHt6ZTOqNPY7DomWNbriX/fuYzkg2jY/eAPM5zD
//+d7/7OvcJkFXIlqDX0ahDGE18LbJt6cSOtWWTaDhGbPonQdZEsIrY7cuWQwLapFzfZ9FlkES8ERL8W
EPteQMLHYFR69coFUNcs5o2AWe+nYW5HKFK7otHkL1IuoKBbZLzzFHMo0qwxWN8zG2eHSpULkGxalmIO
w2qrCvm2eJr/AlwZrlIuYN9XHcuwqKC3JWC3czGqXKm4OVKvXMARTyHTfUlEmWspatxp9D8Ld0ZM/xYw
EeejMnkMv0tdS1DnzkBzv4jT/jy0fm/EzLvhE0p9P7JV7jB3lHHiu6BHhMGmRbkjH3WebTjq245z/nL6
nYFjvnVo8efjzKAOrcONuD1skoNuDNfj8lAFWgZL0OQrphEWoMa1E+oHkZADuHn0K2L8rYDEjhBoCENO
itSXgr2u5TjkycTx/g1koMP5Qb0ccHogj8xEVLlTsaNvIXJ750BjiUZSewhU5LPTIv0N4J1zc76osUTR
ZjWKAuaHvZk44dPijF9HAXk4OaCFyZdNzyIde5zLUGifh5yeOKzoikJ8RzBU7wSE06Us+jQmoMAqIpnM
U6mDXJtaxrDSrUGDdw2O9a/HCTI96suByZuNw57VqKa1MtcybLXPxTo6qWzeHgx+Gbn5tCcCpHdjAnbZ
NmKtNZYwTESJYwnq6WE2EinNZMhNj3izZDXQqKppJJymLfZkZHfHYWlnJNQfghBFIw5rEyA8Jj2ggJdj
Ag769cw0ZGAtP8rYhZH97NLPanbpVxW7/PMAu/atlnSQzNeg0qWB0bkIensSmj3lqHUaf1d4ilipU2LG
LxIzWCUmfSZ9JL2VmBzAMR0PsVHxTbeGmlmleyWMjoXYTKfMpNPWuI3y2oTimE5WfOP1bw1sBz2X/N4E
pNONnm8Oxx6HQTYJbJt6cZOLAzUsrzdeftHxt2kskbLbFpjx/xY3OeWtYKssMUgkUmYQzvzbUDzK+f8W
N6l3lbCk9lCoCMOI5wKCnhIlZgUD9tkNjJuHUefTOYaPKOCNggEl3RILp84FukDCQ9I9CnihYEBxpySP
hNiWL5DURnqmVACxzI3G1aScC8If1IGynvf45pQAAAAASUVORK5CYII=
</value>
</data>
<data name="BTT_DELETE.Image" type="System.Drawing.Bitmap, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
<value>
iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8
YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAVFSURBVEhLjZVrTJNXGMcLQmdHO6Sdt6ygAyYgN+1bS+uE
4ASlxVfSUkFkRMyouILinShGYzTRKCZL9mH7sH3YMj+YzUSXZXNeQOcEFkFhysbVQWlL6dsLvMwtWaBn
/1OKl1A2nuSX8p7znP/z5znnPa+ARntubkRjXNxnd5Ys6b2xbNmn+XL5WxgO80/+T/SoVOo/YmLu9S9d
+uhqfLwRQxEgxD9J44nZLH6Qm/tNT20tGbh4kTw9dMh3Q6NpyJBI3sF0+FRW8OhimHX2kpJB/vx5Ml5f
TwaKitwfJyZWxURGRmF6qkhDXNwn3QcOkKenTpGBkyeJFcmWI0fIdbX6XppEsgIpQYv0aTTvQdzCX7hA
eKzl6+rI+IkTpJdlPd/K5R8hRQrCBHdkMisVh3MyeOwYsSJp+OxZYq2t9d1QqX5OlUgSkPhKEbRlo724
2EqdU/FxiPOHD5NxGPOWlZHGxYs7kaYGEsGt6OhLvRUVZAAFLMB69CixY8HImTPEjnbdVqubU8TiRCT7
i/QoFBuHi4qsY+fOER5mxmGKx7rx/fvJ6O7dpEGl4splsq+RqgNSgWH58mXXlcp7/Tt3+oaQZAPDcDOC
PXGhZc6DB8kdFFktFic9ZJhcm9FoHzt92t8SHo55tJevqSFjEL+rUrkTRKIvIWwCaYBuuCCckckSUaTZ
smMHse/ZQxzV1WRk3z7CoZiH/jfV1b4WjaZjsKDAyh8/7nfMY45HLl9VNSWuVLqTRKJL0KsAyQHxUOAP
4WqpNOlHhaJpqLTU56isJE7gMpuJGwJeFPPu3UvG4HaMCuOZx7NfHO1t0mg86WIxFafOV4L54MVRDYRQ
JZWuvJWe3mzfupU4y8uJC3hMJuKFw1EUHMUGjhYUEO/69cSr0RA3aMrI8K6WSC5j/S5AnYvADPHpEK6J
ikpuSEtrtrHshHvbNuLSaomLYQgnlxNOIiFceDjh5s0jI2Kx73ZsrEcdFUU3tBKkgNfBrOLT4W/Xg/T0
9mEIOqgYcAIq7CcsjPzOMH8bExO/R/6HIBXMSdwf9rIys33dOotDJJopDuiYNT5+opFl+5OiowuwJBLM
TdxrMh3kNmzgHHA/m7gNDAJLcjJ5VFjYuX3t2lVY+p/Xij9cJlMdt2mTyxERMUPcid7TdlHxAdAbGkqe
gO5Vq3zthYW/GdVq2qbZi3h27TrG5eW5gzl3ymRkKC/vH3tCgo+K9wTE20ATeMwwk+1GY+f22YrgKNZx
Wq3HIRb7hV9py6JFpEun+/N0VlZL2+bNw5akJPIYoq3gPvgJNIL2NWsmW43Gjg1pafSCfHHVuysqajmd
zoNjN1N84ULSk5//rIphbiG1Oi8l5f1Ovb6/C72fFm8AN8F10KpSTbYUFrZ9wbJ044UgROA2GKwjCxYE
dU7FKxWKm0g0A7qR0n3Z2e92GAx9Hampz8V/AN+Ba6A9K2ui1WDoRu5yIBQ4jcYud0xMUOdmhrmNpKqA
+PSXSrg/J0eNje1rS031vSx+JSSEtGZmTtxlWRvyckGUoKW0tGSYZV0uiL7k/K89wcWnQ1iTnZ2Bje3F
C0muBsRxbUxe1mqHdCtWfI6cqQKI+TeLio5bt2zxONVqX49O96xGqWzA+Gzi0yE0ZWYqaDtase4XjWby
ilZre0MkqsecHky1CBHCxMZGPjQYvnqq1zsv5+Tg9IXWYJwBYjpPk2YJ4bXi4g9+1ett91mW0yYk0G+B
AcgBPa7P19I/FgAlyA/8zvX1fw3Qo0lPzmbwNgj6ws0D1DH9WNNf+jyXoCbo/f9mAFowYEwg+Bc5Ntw7
FHW1qQAAAABJRU5ErkJggg==
</value>
</data>
<data name="$this.Icon" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
<value>
AAABAAUAEBAAAAEAIABoBAAAVgAAABgYAAABACAAiAkAAL4EAAAgIAAAAQAIAKgIAABGDgAAMDAAAAEA

View File

@@ -8,28 +8,24 @@
' but WITHOUT ANY WARRANTY
Imports PersonalUtilities.Forms.Toolbars
Imports PersonalUtilities.Forms
Public Class SettingsForm : Implements IOkCancelToolbar
Private ReadOnly MyDefs As DefaultFormProps
Public Class SettingsForm
Private WithEvents MyDefs As DefaultFormOptions
Private ReadOnly Property Settings As SiteSettings
Friend Sub New(ByRef s As SiteSettings)
Friend Sub New(ByRef s As SiteSettings, ByRef Design As XML.XmlFile)
InitializeComponent()
MyDefs = New DefaultFormProps
Settings = s
MyDefs = New DefaultFormOptions(Me, Design)
End Sub
Private Sub SettingsForm_Load(sender As Object, e As EventArgs) Handles Me.Load
Try
With MyDefs
.MyViewInitialize(Me, Settings.Design, True)
.AddOkCancelToolbar()
.DelegateClosingChecker()
If Settings.Domains.Count > 0 Then Settings.Domains.ForEach(Sub(d) LIST_DOMAINS.Items.Add(d))
.EndLoaderOperations()
End With
Catch ex As Exception
MyDefs.InvokeLoaderError(ex)
End Try
With MyDefs
.MyViewInitialize(True)
.AddEditToolbar({EditToolbar.ControlItem.Add, EditToolbar.ControlItem.Delete})
.AddOkCancelToolbar()
If Settings.Domains.Count > 0 Then Settings.Domains.ForEach(Sub(d) LIST_DOMAINS.Items.Add(d))
.EndLoaderOperations()
End With
End Sub
Private Sub ToolbarBttOK() Implements IOkCancelToolbar.ToolbarBttOK
Private Sub MyDefs_ButtonOkClick(ByVal Sender As Object, ByVal e As KeyHandleEventArgs) Handles MyDefs.ButtonOkClick
Settings.Domains.Clear()
With LIST_DOMAINS
If .Items.Count > 0 Then
@@ -39,10 +35,7 @@ Public Class SettingsForm : Implements IOkCancelToolbar
Settings.UpdateDomains()
MyDefs.CloseForm()
End Sub
Private Sub ToolbarBttCancel() Implements IOkCancelToolbar.ToolbarBttCancel
MyDefs.CloseForm(Windows.Forms.DialogResult.Cancel)
End Sub
Private Sub BTT_ADD_Click(sender As Object, e As EventArgs) Handles BTT_ADD.Click
Private Sub MyDefs_ButtonAddClick(ByVal Sender As Object, ByVal e As EditToolbarEventArgs) Handles MyDefs.ButtonAddClick
Dim nd$ = InputBoxE("Enter a new domain using the pattern [xvideos.com]:", "New domain")
If Not nd.IsEmptyString Then
If Not LIST_DOMAINS.Items.Contains(nd) Then
@@ -52,11 +45,10 @@ Public Class SettingsForm : Implements IOkCancelToolbar
End If
End If
End Sub
Private Sub BTT_DELETE_Click(sender As Object, e As EventArgs) Handles BTT_DELETE.Click
Private Sub MyDefs_ButtonDeleteClickE(ByVal Sender As Object, ByVal e As EditToolbarEventArgs) Handles MyDefs.ButtonDeleteClickE
If _LatestSelected.ValueBetween(0, LIST_DOMAINS.Items.Count - 1) Then
Dim n$ = LIST_DOMAINS.Items(_LatestSelected)
If MsgBoxE({$"Are you sure you want to delete the [{n}] domain?",
"Removing domains"}, MsgBoxStyle.YesNo) = MsgBoxResult.Yes Then
If MsgBoxE({$"Are you sure you want to delete the [{n}] domain?", "Removing domains"}, MsgBoxStyle.YesNo) = MsgBoxResult.Yes Then
LIST_DOMAINS.Items.RemoveAt(_LatestSelected)
MsgBoxE($"Domain [{n}] removed")
Else

View File

@@ -26,7 +26,7 @@ Public Class SiteSettings : Implements ISiteSettings
Public Property Logger As ILogProvider Implements ISiteSettings.Logger
#Region "M3U8"
Private ReadOnly OS64 As Boolean
Private ReadOnly FfmpegExists As Boolean
Friend ReadOnly FfmpegExists As Boolean
Friend ReadOnly FfmpegFile As SFile
Friend ReadOnly Property UseM3U8 As Boolean
Get
@@ -41,7 +41,6 @@ Public Class SiteSettings : Implements ISiteSettings
Public ReadOnly Property Responser As Response
Private Const DomainsDefault As String = "xvideos.com|xnxx.com"
Private _Initialized As Boolean = False
Friend Design As XmlFile
Public Sub New()
Responser = New Response($"Settings\Responser_{Site}.xml")
With Responser
@@ -78,12 +77,12 @@ Public Class SiteSettings : Implements ISiteSettings
If Not ACheck(SiteDomains.Value) Then SiteDomains.Value = DomainsDefault
Domains.ListAddList(CStr(SiteDomains.Value).Split("|"), LAP.NotContainsOnly, LAP.ClearBeforeAdd)
Domains.ListAddList(DomainsDefault.Split("|"), LAP.NotContainsOnly)
SiteDomains.Value = Domains.ListToString(, "|")
SiteDomains.Value = Domains.ListToString("|")
_DomainsUpdateInProgress = False
End If
End Sub
#Region "Downloading"
Public Function Available(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.Available
Public Function Available(ByVal What As ISiteSettings.Download, ByVal Silent As Boolean) As Boolean Implements ISiteSettings.Available
Return UseM3U8
End Function
Public Function ReadyToDownload(ByVal What As ISiteSettings.Download) As Boolean Implements ISiteSettings.ReadyToDownload
@@ -98,6 +97,10 @@ Public Class SiteSettings : Implements ISiteSettings
Public Sub DownloadDone(ByVal What As ISiteSettings.Download) Implements ISiteSettings.DownloadDone
End Sub
#End Region
Public Sub BeginEdit() Implements ISiteSettings.BeginEdit
End Sub
Public Sub EndEdit() Implements ISiteSettings.EndEdit
End Sub
Public Sub BeginUpdate() Implements ISiteSettings.BeginUpdate
End Sub
Public Sub EndUpdate() Implements ISiteSettings.EndUpdate
@@ -109,10 +112,8 @@ Public Class SiteSettings : Implements ISiteSettings
Public Sub Reset() Implements ISiteSettings.Reset
End Sub
Public Sub OpenSettingsForm() Implements ISiteSettings.OpenSettingsForm
Using f As New SettingsForm(Me)
Design = New XmlFile("Settings\Design_XVIDEOS.xml")
f.ShowDialog()
Design.Dispose()
Using Design As New XmlFile("Settings\Design_XVIDEOS.xml")
Using f As New SettingsForm(Me, Design) : f.ShowDialog() : End Using
End Using
End Sub
Public Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean) Implements ISiteSettings.UserOptions
@@ -182,4 +183,7 @@ Public Class SiteSettings : Implements ISiteSettings
End If
Return Nothing
End Function
Public Function GetUserPostUrl(ByVal UserID As String, ByVal PostID As String) As String Implements ISiteSettings.GetUserPostUrl
Return String.Empty
End Function
End Class

View File

@@ -10,10 +10,12 @@ Imports PersonalUtilities.Functions.RegularExpressions
Imports PersonalUtilities.Functions.XML
Imports PersonalUtilities.Tools.WEB
Imports PersonalUtilities.Tools.WebDocuments.JSON
Imports UStates = SCrawler.Plugin.PluginUserMedia.States
Imports UTypes = SCrawler.Plugin.PluginUserMedia.Types
Public Class UserData : Implements IPluginContentProvider
#Region "Interface declarations"
Public Event ProgressChanged(Count As Integer) Implements IPluginContentProvider.ProgressChanged
Public Event TotalCountChanged(Count As Integer) Implements IPluginContentProvider.TotalCountChanged
Public Event ProgressChanged(ByVal Count As Integer) Implements IPluginContentProvider.ProgressChanged
Public Event TotalCountChanged(ByVal Count As Integer) Implements IPluginContentProvider.TotalCountChanged
Public Property Thrower As IThrower Implements IPluginContentProvider.Thrower
Public Property LogProvider As ILogProvider Implements IPluginContentProvider.LogProvider
Public Property ESettings As ISiteSettings Implements IPluginContentProvider.Settings
@@ -35,7 +37,8 @@ Public Class UserData : Implements IPluginContentProvider
Public Property SeparateVideoFolder As Boolean Implements IPluginContentProvider.SeparateVideoFolder
Public Property DataPath As String Implements IPluginContentProvider.DataPath
Public Property PostsNumberLimit As Integer? Implements IPluginContentProvider.PostsNumberLimit
Public Property PostsDateLimit As Date? Implements IPluginContentProvider.PostsDateLimit
Public Property DownloadDateFrom As Date? Implements IPluginContentProvider.DownloadDateFrom
Public Property DownloadDateTo As Date? Implements IPluginContentProvider.DownloadDateTo
#End Region
#Region "Interface exchange options"
Public Sub ExchangeOptionsSet(ByVal Obj As Object) Implements IPluginContentProvider.ExchangeOptionsSet
@@ -54,14 +57,21 @@ Public Class UserData : Implements IPluginContentProvider
Private Property Responser As Response
Public Sub GetMedia() Implements IPluginContentProvider.GetMedia
Try
If Not Settings.UseM3U8 Then LogProvider.Add("File [ffmpeg.exe] not found") : Exit Sub
If Not Settings.UseM3U8 Then
If Settings.FfmpegExists Then
LogProvider.Add($"XVIDEOS [{Name}]: The plugin only works with x64 OS.")
Else
LogProvider.Add($"XVIDEOS [{Name}]: File [ffmpeg.exe] not found")
End If
Exit Sub
End If
If Not Responser Is Nothing Then Responser.Dispose()
Responser = New Response
Responser.Copy(Settings.Responser)
Dim NextPage% = 0
Dim r$
Dim j As EContainer, jj As EContainer
Dim jj As EContainer
Dim e As ErrorsDescriber = EDP.ThrowException
Dim user$ = Settings.GetUserUrl(Name, False)
Dim p As PluginUserMedia
@@ -71,9 +81,8 @@ Public Class UserData : Implements IPluginContentProvider
Thrower.ThrowAny()
r = Responser.GetResponse($"https://www.xvideos.com/{user}/videos/new/{If(NextPage = 0, String.Empty, NextPage)}",, e)
If Not r.IsEmptyString Then
If Not EnvirSet Then UserExists = True : UserSuspended = False
j = JsonDocument.Parse(r).XmlIfNothing
With j
If Not EnvirSet Then UserExists = True : UserSuspended = False : EnvirSet = True
With JsonDocument.Parse(r).XmlIfNothing
If .Contains("videos") Then
With .Item("videos")
If .Count > 0 Then
@@ -84,12 +93,16 @@ Public Class UserData : Implements IPluginContentProvider
.URL = $"https://www.xvideos.com{jj.Value("u")}"
}
If Not p.PostID.IsEmptyString And Not jj.Value("u").IsEmptyString Then
If Not TempPostsList.Contains(p.PostID) Then TempPostsList.Add(p.PostID) : TempMediaList.Add(p) Else Exit Do
If Not TempPostsList.Contains(p.PostID) Then TempPostsList.Add(p.PostID) : TempMediaList.Add(p) Else .Dispose() : Exit Do
End If
Next
Else
.Dispose()
Exit Do
End If
End With
Else
.Dispose()
Exit Do
End If
.Dispose()
@@ -102,9 +115,7 @@ Public Class UserData : Implements IPluginContentProvider
If TempMediaList.Count > 0 Then
For i% = 0 To TempMediaList.Count - 1
Thrower.ThrowAny()
With TempMediaList(i)
TempMediaList(i) = GetVideoData(.URL, Responser, Settings.DownloadUHD.Value, .PostID, LogProvider)
End With
With TempMediaList(i) : TempMediaList(i) = GetVideoData(.URL, Responser, Settings.DownloadUHD.Value, .PostID, LogProvider) : End With
Next
TempMediaList.RemoveAll(Function(m) m.URL.IsEmptyString)
End If
@@ -146,8 +157,8 @@ Public Class UserData : Implements IPluginContentProvider
Dim t$ = RegexReplace(r, VideoTitleRegex)
r = resp.GetResponse(m,, EDP.ThrowException)
If Not r.IsEmptyString Then
Dim ls As List(Of VSize) = FNF.RegexFields(Of VSize)(r, {M3U8Reparse}, {1, 2})
If ls.ListExists And DownloadUHD Then ls.RemoveAll(Function(v) v.Size > 1080)
Dim ls As List(Of VSize) = RegexFields(Of VSize)(r, {M3U8Reparse}, {1, 2})
If ls.ListExists And Not DownloadUHD Then ls.RemoveAll(Function(v) v.Size > 1080)
If ls.ListExists Then
ls.Sort()
m = $"{appender}/{ls(0).Value}"
@@ -164,7 +175,7 @@ Public Class UserData : Implements IPluginContentProvider
End If
If Not m.IsEmptyString Then
Return New PluginUserMedia With {
.ContentType = PluginUserMedia.Types.m3u8,
.ContentType = UTypes.m3u8,
.PostID = pID,
.URL = m,
.File = $"{t}.mp4",
@@ -194,9 +205,15 @@ Public Class UserData : Implements IPluginContentProvider
m = TempMediaList(i)
f = m.File
f.Path = DefPath
f = M3U8.Download(m.URL, m.SpecialFolder, Settings.FfmpegFile, f, LogProvider)
m.File = f
If Not f.IsEmptyString Then m.DownloadState = PluginUserMedia.States.Downloaded
m.DownloadState = UStates.Tried
Try
f = M3U8.Download(m.URL, m.SpecialFolder, Settings.FfmpegFile, f, LogProvider)
m.File = f
m.DownloadState = UStates.Downloaded
Catch ex As Exception
m.DownloadState = UStates.Missing
m.Attempts += 1
End Try
TempMediaList(i) = m
RaiseEvent ProgressChanged(1)
Next

View File

@@ -1,3 +1,3 @@
[*.vb]
# Modifier preferences
file_header_template = Copyright (C) 2022 Andy\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>
file_header_template = Copyright (C) 2023 Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
@@ -32,6 +32,10 @@ Namespace Plugin.Attributes
Public Property AllowNull As Boolean = True
''' <summary>Offset the control from the left border of the form.<br/>Default: 100</summary>
Public Property LeftOffset As Integer = 100
''' <summary>This control is an information label.<br/>Default: <see langword="False"/></summary>
Public Property IsInformationLabel As Boolean = False
''' <summary>Label text alignment.<br/>Default: <see cref="Drawing.ContentAlignment.TopCenter"/></summary>
Public Property LabelTextAlign As Drawing.ContentAlignment = Drawing.ContentAlignment.TopCenter
''' <summary>This is an authorization property</summary>
Public Property IsAuth As Boolean = False
''' <summary>Initialize a new property option attribute</summary>
@@ -49,6 +53,9 @@ Namespace Plugin.Attributes
ElementName = XMLElementName
End Sub
End Class
''' <summary>Attribute to disable some properties for host use</summary>
<AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class DoNotUse : Inherits Attribute
End Class
''' <summary>Special property updater</summary>
<AttributeUsage(AttributeTargets.Method, AllowMultiple:=True, Inherited:=False)> Public NotInheritable Class PropertyUpdater : Inherits Attribute
Public ReadOnly Name As String
@@ -93,6 +100,8 @@ Namespace Plugin.Attributes
''' <see langword="False"/> - only for conversion
''' </summary>
Public FieldsChecker As Boolean = False
''' <summary>Interaction with changing text field. Default: <see langword="False"/></summary>
Public Interaction As Boolean = False
''' <summary>Initialize a new Provider attribute. <see cref="IFormatProvider"/> is only allowed</summary>
''' <param name="PropertyName">The name of the property for which this provider is used</param>
Public Sub New(ByVal PropertyName As String)
@@ -125,13 +134,26 @@ Namespace Plugin.Attributes
''' Predefined task counter.<br/>
''' <see cref="TaskCounter"/> will take precedence if it is defined.
''' </param>
Public Sub New(Optional ByVal JobsCount As Integer = -1)
TasksCount = JobsCount
Public Sub New(Optional ByVal TasksCount As Integer = -1)
Me.TasksCount = TasksCount
End Sub
End Class
''' <summary>A property attribute that specifies how many users should be downloaded at the same time in one thread</summary>
<AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class TaskCounter : Inherits Attribute
End Class
''' <remarks>
''' This attribute cannot be combined with <see cref="SeparatedTasks"/>.
''' If set to <see cref="SeparatedTasks"/>, this attribute will be ignored
''' </remarks>
''' <inheritdoc cref="SeparatedTasks"/>
<AttributeUsage(AttributeTargets.Class, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class TaskGroup : Inherits Attribute
Public ReadOnly Name As String
''' <summary>Initialize a new TaskGroup attribute.</summary>
''' <param name="Name">Group name</param>
Public Sub New(ByVal Name As String)
Me.Name = Name
End Sub
End Class
''' <summary>This attribute indicates that the plugin has a SavedPosts environment</summary>
<AttributeUsage(AttributeTargets.Class, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class SavedPosts : Inherits Attribute
End Class

View File

@@ -0,0 +1,34 @@
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
' (at your option) any later version.
'
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Namespace Plugin
Public Interface IDownloadableMedia : Inherits IUserMedia, IDisposable
Event CheckedChange As EventHandler
Event ThumbnailChanged As EventHandler
Event StateChanged As EventHandler
ReadOnly Property SiteIcon As Drawing.Image
ReadOnly Property Site As String
ReadOnly Property SiteKey As String
Property ThumbnailUrl As String
Property ThumbnailFile As String
Property Title As String
Property Size As Integer
Property Duration As TimeSpan
Property Progress As Object
ReadOnly Property HasError As Boolean
ReadOnly Property Exists As Boolean
Property Checked As Boolean
Property Instance As IPluginContentProvider
Sub Download(ByVal UseCookies As Boolean, ByVal Token As Threading.CancellationToken)
Sub Delete(ByVal RemoveFiles As Boolean)
Sub Load(ByVal File As String)
Sub Save()
Overloads Function ToString() As String
Overloads Function ToString(ByVal ForMediaItem As Boolean) As String
End Interface
End Namespace

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
@@ -8,8 +8,10 @@
' but WITHOUT ANY WARRANTY
Namespace Plugin
Public Interface IPluginContentProvider : Inherits IDisposable
Event ProgressChanged(ByVal Count As Integer)
Event TotalCountChanged(ByVal Count As Integer)
Event ProgressChanged(ByVal Value As Integer)
Event ProgressMaximumChanged(ByVal Value As Integer, ByVal Add As Boolean)
Event ProgressPreChanged As ProgressChangedEventHandler
Event ProgressPreMaximumChanged As ProgressMaximumChangedEventHandler
Property Thrower As IThrower
Property LogProvider As ILogProvider
Property Settings As ISiteSettings
@@ -17,21 +19,23 @@ Namespace Plugin
Property ID As String
Property ParseUserMediaOnly As Boolean
Property UserDescription As String
Property ExistingContentList As List(Of PluginUserMedia)
Property ExistingContentList As List(Of IUserMedia)
Property TempPostsList As List(Of String)
Property TempMediaList As List(Of PluginUserMedia)
Property TempMediaList As List(Of IUserMedia)
Property UserExists As Boolean
Property UserSuspended As Boolean
Property IsSavedPosts As Boolean
Property SeparateVideoFolder As Boolean
Property DataPath As String
Property PostsNumberLimit As Integer?
Property PostsDateLimit As Date?
Property DownloadDateFrom As Date?
Property DownloadDateTo As Date?
Function ExchangeOptionsGet() As Object
Sub ExchangeOptionsSet(ByVal Obj As Object)
Sub XmlFieldsSet(ByVal Fields As List(Of KeyValuePair(Of String, String)))
Function XmlFieldsGet() As List(Of KeyValuePair(Of String, String))
Sub GetMedia()
Sub Download()
Sub GetMedia(ByVal Token As Threading.CancellationToken)
Sub Download(ByVal Token As Threading.CancellationToken)
Sub DownloadSingleObject(ByVal Data As IDownloadableMedia, ByVal Token As Threading.CancellationToken)
End Interface
End Namespace

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
@@ -12,17 +12,18 @@ Namespace Plugin
Enum Download As Integer
Main = 0
SavedPosts = 1
Channel = 2
SingleObject = 2
End Enum
ReadOnly Property Icon As Icon
ReadOnly Property Image As Image
ReadOnly Property Site As String
Property Logger As ILogProvider
Function GetUserUrl(ByVal UserName As String, ByVal Channel As Boolean) As String
Function GetUserUrl(ByVal User As IPluginContentProvider) As String
Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
Function IsMyImageVideo(ByVal URL As String) As ExchangeOptions
Function GetSpecialData(ByVal URL As String, ByVal Path As String, ByVal AskForPath As Boolean) As IEnumerable(Of PluginUserMedia)
Function GetInstance(ByVal What As Download) As IPluginContentProvider
Function GetSingleMediaInstance(ByVal URL As String, ByVal OutputFile As String) As IDownloadableMedia
Function GetUserPostUrl(ByVal User As IPluginContentProvider, ByVal Media As IUserMedia) As String
#Region "XML Support"
Sub Load(ByVal XMLValues As IEnumerable(Of KeyValuePair(Of String, String)))
#End Region
@@ -31,9 +32,11 @@ Namespace Plugin
Sub EndInit()
Sub BeginUpdate()
Sub EndUpdate()
Sub BeginEdit()
Sub EndEdit()
#End Region
#Region "Site availability"
Function Available(ByVal What As Download) As Boolean
Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean
Function ReadyToDownload(ByVal What As Download) As Boolean
#End Region
#Region "Downloading"

View File

@@ -13,7 +13,7 @@ Imports System.Runtime.InteropServices
<Assembly: AssemblyDescription("Plugin provider for SCrawler")>
<Assembly: AssemblyCompany("AndyProgram")>
<Assembly: AssemblyProduct("SCrawler.PluginProvider")>
<Assembly: AssemblyCopyright("Copyright © 2022")>
<Assembly: AssemblyCopyright("Copyright © 2023")>
<Assembly: AssemblyTrademark("AndyProgram")>
<Assembly: ComVisible(False)>
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("3.0.0.2")>
<Assembly: AssemblyFileVersion("3.0.0.2")>
<Assembly: AssemblyVersion("2023.5.12.0")>
<Assembly: AssemblyFileVersion("2023.5.12.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
@@ -6,7 +6,8 @@
'
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Namespace Plugin.Attributes
<AttributeUsage(AttributeTargets.Class, AllowMultiple:=False, Inherited:=False)> Friend NotInheritable Class UseClassAsIs : Inherits Attribute
End Class
Namespace Plugin
Public Interface IPropertyProvider : Inherits IFormatProvider
Property PropertyName As String
End Interface
End Namespace

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
@@ -11,15 +11,11 @@ Namespace Plugin
Public UserName As String
Public SiteName As String
Public HostKey As String
Public IsChannel As Boolean
Public Exists As Boolean
Public Sub New(ByVal Site As String, ByVal _Name As String)
UserName = _Name
Public Sub New(ByVal Site As String, ByVal Name As String)
UserName = Name
SiteName = Site
End Sub
Public Sub New(ByVal Site As String, ByVal _Name As String, ByVal _IsChannel As Boolean)
Me.New(Site, _Name)
IsChannel = _IsChannel
Exists = Not String.IsNullOrEmpty(Name) And Not String.IsNullOrWhiteSpace(Name)
End Sub
End Structure
End Namespace

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
@@ -7,24 +7,49 @@
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Namespace Plugin
Public Structure PluginUserMedia
Enum Types As Integer
Undefined = 0
[Picture] = 1
[Video] = 2
[Text] = 3
VideoPre = 10
GIF = 50
m3u8 = 100
End Enum
Enum States As Integer : Unknown = 0 : Tried = 1 : Downloaded = 2 : Skipped = 3 : End Enum
Public ContentType As Integer
Public URL As String
Public MD5 As String
Public File As String
Public DownloadState As Integer
Public PostID As String
Public PostDate As Date?
Public SpecialFolder As String
Public Delegate Sub ProgressChange(ByVal Value As Double?, ByVal Maximum As Double?, ByVal Information As String)
Public Enum UserMediaTypes As Integer
Undefined = 0
Picture = 1
Video = 2
Audio = 200
Text = 4
VideoPre = 10
AudioPre = 215
GIF = 50
m3u8 = 100
End Enum
Public Enum UserMediaStates As Integer
Unknown = 0
Tried = 1
Downloaded = 2
Skipped = 3
Missing = 4
End Enum
Public Structure PluginUserMedia : Implements IUserMedia
Public Property ContentType As UserMediaTypes Implements IUserMedia.ContentType
Public Property URL As String Implements IUserMedia.URL
Public Property URL_BASE As String Implements IUserMedia.URL_BASE
Public Property MD5 As String Implements IUserMedia.MD5
Public Property File As String Implements IUserMedia.File
Public Property DownloadState As UserMediaStates Implements IUserMedia.DownloadState
Public Property PostID As String Implements IUserMedia.PostID
Public Property PostDate As Date? Implements IUserMedia.PostDate
Public Property SpecialFolder As String Implements IUserMedia.SpecialFolder
Public Property Attempts As Integer Implements IUserMedia.Attempts
Public Property [Object] As Object Implements IUserMedia.Object
End Structure
Public Interface IUserMedia
Property ContentType As UserMediaTypes
Property URL As String
Property URL_BASE As String
Property MD5 As String
Property File As String
Property DownloadState As UserMediaStates
Property PostID As String
Property PostDate As Date?
Property SpecialFolder As String
Property Attempts As Integer
Property [Object] As Object
End Interface
End Namespace

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
@@ -10,9 +10,9 @@ Namespace Plugin
Public Structure PropertyData
Public ReadOnly Name As String
Public ReadOnly Value As Object
Public Sub New(ByVal _Name As String, ByVal _Value As Object)
Name = _Name
Value = _Value
Public Sub New(ByVal Name As String, ByVal Value As Object)
Me.Name = Name
Me.Value = Value
End Sub
End Structure
End Namespace

View File

@@ -1,4 +1,4 @@
' Copyright (C) 2022 Andy
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or

View File

@@ -102,6 +102,8 @@
</ItemGroup>
<ItemGroup>
<Compile Include="Attributes\Attributes.vb" />
<Compile Include="Interfaces\IDownloadableMedia.vb" />
<Compile Include="ObjectInterfaces\IPropertyProvider.vb" />
<Compile Include="Objects\ExchangeOptions.vb" />
<Compile Include="ObjectInterfaces\ILogProvider.vb" />
<Compile Include="Interfaces\IPluginContentProvider.vb" />

View File

@@ -0,0 +1,3 @@
[*.vb]
# Modifier preferences
file_header_template = Copyright (C) 2023 Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>

View File

@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1" />
</startup>
</configuration>

View File

@@ -0,0 +1,30 @@
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
' (at your option) any later version.
'
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Namespace API.YouTube.Attributes
<AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)>
Public Class GridVisibleAttribute : Inherits Attribute
Private ReadOnly NonAppMode As Boolean = True
Public Sub New()
End Sub
Public Sub New(ByVal NonAppMode As Boolean)
Me.NonAppMode = NonAppMode
End Sub
Public Overrides Function Equals(ByVal Obj As Object) As Boolean
If Not Obj Is Nothing AndAlso TypeOf Obj Is GridVisibleAttribute Then
If NonAppMode Then
Return DirectCast(Obj, GridVisibleAttribute).NonAppMode
Else
Return True
End If
Else
Return False
End If
End Function
End Class
End Namespace

Some files were not shown because too many files have changed in this diff Show More