Compare commits

...

120 Commits

Author SHA1 Message Date
Andy
24ad338c60 2025.6.12.0
YT
MainModShared: fix environment output
YouTubeMediaContainerBase: fix 'm3u8' audio formats

SCrawler
UserDataBase: text downloading with saved posts; update 'ID' property (handle '_ForceSaveUserInfo')
API.Bluesky: data is not downloaded
API.Reddit: update 'RedditViewExchange'; set base inheritance; inherit default settings for new users
API.ALL: update functions with property 'ID'
2025-06-12 20:29:59 +03:00
Andy
ff0c4587eb 2025.6.1.0
PluginProvider
IUserMedia, PluginUserMedia: add properties 'PostText', 'PostTextFile', 'PostTextFileSpecialFolder'

YT
YouTubeFunctions: update 'Info_GetUrlType' and 'StandardizeURL' functions: add youtu.be domain
YouTubeSettings: add 'FILTER' property
Add classes 'FilterForm', 'YTDataFilter'
VideoListForm: add filters; update 'LoadData' and 'RemoveControls' functions; add hotkey 'Ctrl+F5' for refresh
YouTubeMediaContainerBase: add support for new interface properties
Minor bugs

SCrawler
DeclaredNames: add new names
EditorExchangeOptionsBase, IUserData, SiteSettingsBase, UserMedia, UserDataBase: add support for text downloading

Sites Bluesky, Instagram, OnlyFans, Reddit, ThreadsNet, Twitter: add support for text downloading
Sites Facebook, JustForFans, LPSG, Mastodon, Pinterest, PornHub, Redgifs, ThisVid, TikTok, Xhamster, XVIDEOS, YouTube (STD): disable text downloading

UserDataBase: add 'ToStringExt' functions

API.Instagram: add 'SleepTimerRequestsNextProfile' property
API.OnlyFans: update 'DynamicRules'; fix incorrect posts opening (update 'GetUserPostUrl' function); fix limited download ('DownloadTopCount')
API.Reddit: fix post date provider; add 'Best' and 'Rising' view modes; fix request (data is not downloading); set 'BearerTokenUseCurl' to 'False' by default
API.ThreadsNet: change domain from 'net' to 'com'; fix data downloading
API.TikTok: add downloading of avatar, site name and description
API.Twitter: fix JSON error; add debug options; fix downloading
API.Xhamster: add folder 'Photo' for albums

Feed: add filters; update move/copy algo; add the ability to show test posts; update table rendering; add new 'MediaItem' handlers
FeedMedia: add text options; update 'DeleteFile' function
FeedMoveCopyTo: add text option

VideoDownloaderForm: disable filter button

GlobalSettingsForm: add 'FeedShowTextPosts' and 'FeedShowTextPostsAlwaysMove' options
SettingsCLS: add feed text properties
UserImage: add 'CreateImageFromText' function
UserInfo: update 'Equals' function

Add classes: 'FeedFilter', 'FeedFilterCollection', 'FeedFilterForm'

Minor bugs and improvements
2025-06-01 19:01:26 +03:00
Andy
fff63d0a9f 2025.3.17.0
API.SiteSettingsBase: fix incorrect class initializer
API.UserDataBase: add all objects to xml (STD)
API.Facebook: fix downloading reels from noname profiles
API.Pinterest: remove 'UserOptions' overrides (SiteSettings); add 'PwsHeader' to 'GetBoards'
API.PornHub: fix 'UpdateUserOptions' function ('NameTrue')
API.Threads: fix 'pinned' posts
API.TikTok: add photos download
2025-03-17 16:23:41 +03:00
Andy
2f838929cc 2025.2.25.0
PluginProvider
IPluginContentProvider: add 'NameTrue' property

YT
YouTubeSettings: remove the 'CreateDescriptionFiles_AddUploadDate' property
PlayListParserForm: add 'RDAMP' as default value when initializing form
YouTubeMediaContainerBase: fix line breaks

SCrawler
API.Base: add 'EditorExchangeOptionsBase' class
API.Base.GDL: move functions to Pinterest.UserData
API.Base.IUserData: add 'NameTrue' property
API.Base.SiteSettingsBase: update the 'GetUserUrl' function to use the 'NameTrue' property; update 'UserOptions' function
API.Base.UserDataBase: add 'NameTrue' property; add 'SimpleDownloadAvatar' function to get rid of the  same functions of other classes; Update 'UserDescriptionUpdate' function
API.Base.InternalSettingsForm: calculate max offset

ADD API.Bluesky

API.Facebook: add 'Reels' downloads; fix video downloading
API.Instagram: add inheritance 'EditorExchangeOptionsBase'; remove 'GetUserUrl' function from 'SiteSettings'; update 'UserData' class to new environment; add saving 'heic' file along with 'jpg'
API.LPSG.UserData: simplify 403 error
API.Mastodon: update classes to new environment
API.OnlyFans: add 'AppTokenDefault'; disable cookies update; update 'UserData' class to new environment
API.Pinterest: add sub-boards downloading; update download functions
API.PornHub: fix photo & video downloading; remove 'ModelHub' support
API.Reddit: fix token update; update 'UserData' class to new environment
API.ThisVid: update 'UserData' class to new environment
API.ThreadsNet: fix data download; update classes to new environment; fix 'UserID' extraction; add the ability to manually change the UserName
API.TikTok: update classes to new environment
API.Twitter: update classes to new environment; add sleep timers to fully download large profiles; add 'CookiesUpdate' hidden property
API.Xhamster: update classes to new environment
API.XVIDEOS: update classes to new environment

Feed: add the ability to invert selection; open post URL when double-clicking on subscription image
FeedSpecialCollection: update 'FeedsComparer'
GlobalSettingsForm: remove 'UserAgent' from the 'Basis' tab
UserDataHost: update class to new environment
SettingsCLS: set the 'UserSiteNameAsFriendly' property to 'True' by default; disable 'DownDetector'; add 'UsersListProtected' property
2025-02-25 19:47:33 +03:00
Andy
4d74f5204b 2025.1.12.0
YT
YouTubeSettings: add 'FileAddChannelToFileName' property
YouTubeMediaContainerBase: add channel name and video URL to info file; add channel name to file name

SCrawler
DownDetector: fix 403 error; add 'IDownDetector' interface and 'Checker' class; create an isolated environment
API.Instagram: update 'SiteSettings' to the new 'DownDetector' environment; make 'PostKV' public; add static function 'LoadSavePostsKV'
API.OnlyFans: add 'EnableCookiesUpdate' hidden property; add support for DRM keys; add the ability to disable cookie updates
API.Pinterest: add 'x-pinterest-pws-handler' header
API.Reddit: update 'SiteSettings' to the new 'DownDetector' environment
API.ThisVid: fix subscription videos images
API.Threads: change 'heic' extension to 'jpg'
API.Twitter: add broadcasts download
API.Xhamster: fix absolute M3U8 URLs
API.YouTube: add support of personal API instances ('YouTube-operational-API') for download communities
SiteEditorForm: add 'Ctrl+Enter' hotkey to force save settings, ignoring  requirements
PluginsEnvironment.Attributes: add 'UseDownDetectorAttribute' attribute
SettingsHost: update to the new 'DownDetector' environment; add 'AvailableDownDetector' property
SettingsHostCollection: update to the new 'DownDetector' environment; minor bugs in multiprofile
SettingsCLS: add 'DownDetectorEnabled' property
2025-01-12 23:16:57 +03:00
Andy
b42832719f 2024.11.21.0
API.Instagram: code refactoring (settings); add setting to skip errors; add 'ForceUpdateUserName' and 'ForceUpdateUserInfo' properties; add 'IgnoreStoriesDownloadingErrors' to the settings; improve username update algorithm
API.OnlyFans: add 'UpdateRules401' property; update the code to handle error 401
API.YouTube: 404 error handling (community)
UserDataBase: raise event to update user in exceptions; add extra buttons for special download (limited and dated)
UserDataBind: extra buttons (UserDataBase)
UserCreatorForm: fix network paths
GlobalSettingsForm, MainFrame, SettingsCLS: add ability to change the feed opening shortcut
MainFrame: update button captions, update 'DownloadSelectedUser' function
2024-11-21 17:50:19 +03:00
Andy
aedcebc781 2024.10.24.0
YT
YouTubeSettings: add 'DefaultVideoAllowWebm' and 'DefaultAudioEmbedThumbnail_Cover' settings
YouTubeMediaContainerBase: change cover selection for music download; fix adding incorrect playlist lines; allow 'webm' formats is there are no 'mp4' formats via http protocol

SCrawler
DeclaredNames: add new names
UserDataBase: add '_ForceSaveUserInfoOnException' field  and 'UpdateUserInformation_Ex' function to update user info on exception; clear '_MD5List' when clearing data and/or history
API.Instagram: add manual 'UserName' changing; mark user as non-existent if user ID cannot be obtained
API.Twitter: add manual 'UserName' changing
API.Mastodon: bypass inherited property
API.Reddit: fix incorrect UNIX date parsing
DownloadFeedForm: add exception handling to the 'RefillAfterDelete' function
MainFrame: add 'MENU_INFO_USER_SEARCH' to the 'Info' menu
SettingsHostCollection: fix a bug when changing data paths
2024-10-24 19:18:29 +03:00
Andy
00a06d3e9a 2024.9.2.0
Instagram: add options to enable/disable image extraction from video
OnlyFans: update to the changed API
YouTube: videos are parsed from the 'featured', not from the 'videos' page
Feed: add prompt before moving entire feed/session
MainFrame: add 'Alt+U' and 'Ctrl+U' to open the user search form
UserImage: user image creation update
2024-09-02 18:22:11 +03:00
Andy
2055461829 Update FAQ.md 2024-08-14 13:12:25 +03:00
Andy
723155e20c Update FAQ.md 2024-08-14 12:27:05 +03:00
Andy
effaa3b65b Update FAQ 2024-08-14 11:45:00 +03:00
Andy
e285de10f6 2024.8.10.0
YT
Fix bug when video is parsed using cookies but not downloaded

SCrawler
Feed: add a button to open file folder
2024-08-10 13:36:31 +03:00
Andy
26db0e3e24 2024.8.1.0
Feed: add the ability to set PostUrl for data when moving a file and/or adding to a feed
TDownloader: add 'PostUrl' property to 'UserMediaD'
2024-08-01 20:41:51 +03:00
Andy
0b0933b6f0 2024.7.24.0
YT
YouTubeSettings: add 'DefaultVideoConvertNonAVC' property; fix 'OpenFolderInOtherProgram' property serialize and reset; add 'MusicPlaylistCreate_CreationMode' property
Update the 'CleanFileName' function to remove line breaks
Add the ability to convert non-avc codecs to avc
Add 'M3U8CreationMode' enum

SCrawler
API.OnlyFans: fix incorrect delimiter (rules parsing)
API.Threads: add saved posts downloading
Feed: add the hotkeys 'Esc' and 'Ctrl+W' to close the form; add the ability to search for missing files in special feeds
Scheduler: add the ability to execute a script after the scheduler plan is executed
Settings: add enable/disable the use of the 'Esc' to close the feed; add 'AutomationScript' and 'AutomationScript_ExcludeManual' properties
MainFrame: add the hotkey 'Ctrl+F' to show the feed; change the hotkey from 'Ctrl+F' to 'Alt+F' to show the search form
2024-07-24 23:40:53 +03:00
Andy
3ce9c55575 Update OF DynamicRules 2024-07-08 22:35:28 +03:00
Andy
ef36a11566 Update OF DynamicRules 2024-07-07 09:36:10 +03:00
Andy
dea14d35af 2024.6.25.0
API.OnlyFans: new dynamic rules updating algo
API.Instagram: update settings
Feed: add ability to set the last session as the current one; wrong marking data as saved posts when moving a file
2024-06-25 11:46:33 +03:00
Andy
744698c99e 2024.6.13.0
Remove compatibility of settings of older versions
2024-06-13 12:16:38 +03:00
Andy
aef4ce1c8f 2024.6.10.0
YT
YouTubeSettings: add 'VideoPlaylist_AddExtractedMP3' property
MediaItem: improve visualization of height and bitrate
YouTubeMediaContainerBase: fix bugs on default post-processing formats; add 'HeightBase' and 'BitrateBase' properties; add extracted MP3 to playlist
VideoListForm: add 'UpdateLogButton' handlers

SCrawler
Feed: add settings to show/hide site name and file type from media title; add move/copy files of a loaded feed/session to another location; add the ability to reset current session
DownloadFeedForm: when moving saved posts files without replacing the profile, some data is lost
FeedVideo: add double-click handler to open video in external player
API.Instagram: update settings
2024-06-10 08:40:40 +03:00
Andy
93ea2a55ac 2024.6.6.0
YT
VideoOptionsForm: file path is cleared when the cancel button is clicked (browse button); remove the context menu when the right clicking on browse button; add 'ButtonRC' class

SCrawler
API.OnlyFans: add check config to the SiteSettings; update config; add 'Keydb_Api' property; reset 'LastDateUpdated' when rules change; add support 'prefix/suffix' and 'start/end' to support other rules formats
DownloadFeedForm: update 'BTT_CURR_SESSION_SET_Click' function
TDownloader: update 'FilesLoadLastSession' function
2024-06-06 05:49:50 +03:00
Andy
2ae8c3acfc 2024.6.4.0
API.Twitter: add communities downloading, change post opening URL
Feed: add the ability to select one of the download sessions and set it as the current session
2024-06-04 03:28:05 +03:00
Andy
53dcb3e2c6 2024.6.2.0
YT
Add 'FileAddDateToFileName', 'FileAddDateToFileName_VideoForm' and 'FileAddDateToFileName_VideoList' properties
Update 'YouTubeMediaContainerBase', 'VideoOptionsForm' and 'MediaItem' (new options)

SCrawler
API.Instagram: update settings values
API.Reddit: fix 'ReparseMissing' function (remove bearer token)
2024-06-02 01:19:41 +03:00
Andy
ca384e54d6 2024.5.29.0
YT
Trim urls to get rid of 'cr' & 'lf'
Get the correct 'music' url for 'url' files

SCrawler
Remove the no longer needed 'MainFrameObj.UpdateLogButton' from the classes
ProfileSaved: swap the 'ReadyToDownload' and 'Available' checks; remove the 'DownloadStarted' and 'DownloadDone' calls because they are called in the root function
API.Instagram: improve availability checking
API.Twitter: fix deleting user directory when redownloading missing posts
AutoDownloader: improve statuses; move the check thread to the scheduler; add highlighting of scheduler plans (working, stopped, pending, etc.); replace 'ListBox' with 'ListView'; highlight undownloaded plans in gray
2024-05-29 02:45:37 +03:00
Andy
5a1b5c828a 2024.5.25.0
Move files to another directory
2024-05-25 10:06:50 +03:00
Andy
22c59b41f0 2024.5.19.0
YT
YouTubeSettings: add 'CreateDescriptionFiles_AddUploadDate' and 'CreateDescriptionFiles_CreateWithNoDescription' properties
YouTubeMediaContainerBase: add upload date to description

SCrawler
API.YouTube: 'YTSettings_Internal' is not saved when changed
2024-05-19 22:29:16 +03:00
Andy
444b3521eb 2024.5.18.0
YT
YouTubeSettings: add 'DefaultVideoHighlightFPS_H' and 'DefaultVideoHighlightFPS_L' properties
VideoOption: highlight frame rates higher/lower than this value

SCrawler
SiteSettingsBase: add 'UserAgentDefault' property
API.Facebook, API.Instagram, API.Mastodon, API.OnlyFans, API.ThreadsNet, API.Twitter: add categories
API.Instagram.SiteSettings: add 'DownDetector' validation; remove wrong header
API.Instagram.UserData: fix incorrect definition of pinned posts; add 'DefaultParser_Pinned' and 'DefaultParser_SkipPost' func (for Threads)
API.Threads: fix pinned posts processing
API.Reddit: add 429 bypass; change the naming method of video files (hosted on Reddit) to the 'YYYYMMDD_HHMMSS' pattern; add 'UserAgent' property
API.RedGifs: hide credential controls
API.Twitter: add 'Likes' downloading; change domain from twitter.com to x.com;
API.OnlyFans: set '_AllowUserAgentUpdate' to false
SiteEditorForm: group options by category
GroupListForm: enable 'OK' if it is filter
DownloadGroup: add 'FilterShowAllUsers' property
PropertyValueHost: add 'Category' property
MainFrame: the 'ALL' filter isn't unchecked when loading a filter from a saved one
Update user paths when global paths change
Scheduler: add the ability to clone the scheduler

PluginProvider
PropertyOption attribute: set category name when `IsAuth = True`
ISiteSettings: add 'UserAgentDefault' property
2024-05-18 01:17:29 +03:00
Andy
ec2266f1bf 2024.5.4.0
YT
remove dots from the end of the file name; add a setting to remove specific characters

SCrawler
API.Instagram: simplify the 'Connection closed' error
API.Reddit: update token refresh request; add 'BearerTokenUseCurl' hidden property
API.Threads: fix frong header name ('dnt'); update 'UpdateCredentials' function
AutoDownloader: change 'IndexOutOfRangeException' to 'Exception' in the 'Download' function
TDownloader: fix 'FilesUpdatePendingUsers' function (freeze)
UserSearchForm: add 'FriendlyName' to search results
2024-05-04 07:04:26 +03:00
Andy
7d9255c916 2024.4.26.0
Add 'CookieValueExtractorAttribute' and the ability to immediately populate fields with values that can be extracted from cookies
Feed: add the ability to load the last session of the current day (if it exists) as the current session after restarting SCrawler
UserSearchForm: include friendly name matches in search result
API.Xhamster: saved posts aren't downloading
2024-04-26 17:18:31 +03:00
Andy
5b5857e31d 2024.4.14.0
Delete old notes and comments
API.Facebook: add app-id extraction from page; remove app-id from site requirements; update tokens parsing; update tokens regex
API.Instagram: add default function to parse tokens
2024-04-14 07:44:05 +03:00
Andy
46372ec9fb 2024.4.13.0
YT
Add subtitles to information about downloaded files

PluginProvider
IPluginContentProvider: add 'ResetHistoryData' function

SCrawler
UserDataBase: call 'UpdateUsersList' when 'UpdateUserInformation' with argument 'OnlyDiff'; implement 'ResetHistoryData' function; set 'LastUpdated' to null and 'UpdateUserInformation' when erasing history data
API.Instagram: set 'FirstLoadingDone' to false when erasing history data; fix broken saved posts downloading
API.TikTok: set 'LastDownloadDate' to null when erasing history data
API.YouTube: set last download dates to null when erasing history data
GroupUsersViewer: add the number of users, object type and object name to the form title; add 'F1' to help hint; add '(Alt+)F3' to edit user
AutoDownloaderEditorForm, SchedulerEditorForm, GroupEditorForm, GroupListForm: update to 'GroupUsersViewer'
MainFrame: make 'EditSelectedUser' friend
UserDataHost: implement 'ResetHistoryData' function
SettingsCLS: add 'OnlyDiff' to the 'UpdateUsersList' function
UserInfo: add 'ExactEquals' shared function
2024-04-13 10:10:27 +03:00
Andy
7296fda977 2024.4.10.0 2024-04-10 06:25:39 +03:00
Andy
5f90bf6a99 2024.4.8.0
YT
MusicPlaylistsForm, VideoOptionsForm: add audio bitrate option
MediaItem: update type icon; update confirmation dialog for deleting non-single object
Track: update extension
PlayList: update 'ToString' information for 'MediaItem'
YouTubeMediaContainerBase: add size recalculation; add audio bitrate change; embed thumbnail in the extracted 'mp3' as cover art; update 'DownloadCommand' function; include elements' files in XML for non-single items; update 'Delete' function; update files handling; include generated playlists and cover file in the file list
YouTubeSettings: add properties 'DefaultAudioEmbedThumbnail_ExtractedFiles', 'DefaultAudioBitrate', 'DefaultAudioBitrate_crf'
Exclude 'drc' from parsing results
Fix incorrect file reference when the yt-dlp.exe has a different name

SCrawler
Base.Declarations: hide 'TokenRefreshIntervalProvider' error
Base.DeclaredNames: remove 'Header_FB_FRIENDLY_NAME' const (use 'API.Instagram.UserData.GQL')
Base.M3U8Base: add 'SkipBroken' argument
Base.UserDataBase: add size recalculation (STD)
Base.SiteSettingsBase: add 'SettingsVersion' property
TDownloader: delete 'RenameOldFileNames' function
SiteEditorForm: remove begin/end update of global settings when updating
MainFrame: update 'BTT_DOWN_SPEC' tooltip
SettingsHostCollection, SettingsHost: move site settings to a personal setting file (delete these settings from the global settings file)
DownloadGroupCollection: remove data update during initialization and reindexing
SettingsCLS: add 'SettingsVersion' property
Add hidden controls
API.JustForFans: change m3u8 parsing and downloading algo; remove 'CancellationToken' from m3u8 (replace with 'IThrower')
API.Facebook: add option 'RequestsWaitTimer_Any'; add internal option 'DownloadData_Impl'; update GDL names and tokens references; add wait timers
API.Threads: add option 'RequestsWaitTimer_Any'; add internal option 'DownloadData_Impl'; update GDL names and tokens references; add wait timers
API.Instagram: ADD 'GDL' SUPPORT; add 'UpdateWwwClaim' to 'Declarations.UpdateResponser' and 'UserData'; add additional 'HH_IG_WWW_CLAIM' properties; add 'RequestsWaitTimer_Any' property; add tooltips for timer controls; update 'LastRequests' environment; update information about requests on the label in the settings form; update reels downloading function
2024-04-09 21:39:09 +03:00
Andy
718eccc3c3 2024.3.31.0
Plugins
ISiteSettings: add 'CMDEncoding', 'EnvironmentPrograms' properties; add 'EnvironmentProgramsUpdated' function

SCrawler
GlobalSettingsForm: add the ability to find the program environment
SettingsCLS: update 'ProgramFile' class; add 'EnvironmentProgramsList' property and 'UpdateEnvironmentPrograms' function
Add the ability to display group users
Add 'GroupUsersViewer'
2024-03-31 13:18:20 +03:00
Andy
efa09fb457 2024.3.30.0
UPDATE DOWNLOAD GROUP ENVIRONMENT

Add the ability to filter users who have been (not)downloaded in the last x days.
DownloadedInfoForm: fix possible bug
Feed: fix scrolling bug
IUserData, UserDataBase, UserDataBind: remove 'FitToAddParams'
UserDataBase: update 'GetLVIGroup' function; wrong decision to set 'LastUpdated' date
AutoDownloader: remove 'All' and 'Default' options
SettingsCLS: refactoring the code and XML file
2024-03-30 15:42:30 +03:00
Andy
b252d32a7e 2024.3.26.0
API.Instagram: extract image from video
API.Reddit: add 'TryImage' bypass
Feed: add hotkeys: 'Home', 'End', 'Up', 'Page Up', 'Down', 'Page Down'; fix form deactivating; add ability to save/load view
MainFrame: update the background picture if it has changed
2024-03-26 14:13:01 +03:00
Andy
34cd510507 2024.3.24.0
YT
YouTubeMediaContainerBase: add item index to playlist row only if it is > 0

SCrawler
API.Instagram: handle 'JsonNull' for saved posts
GlobalSettingsForm: move the 'GroupUser' option to MainFrame
MainFrame: add 'GroupUser' option and 'ViewFilter' options
SettingsHostCollection: update the 'Silent' parameter for single instance
2024-03-24 02:56:08 +03:00
Andy
f5dd791941 2024.3.22.0
TDownloader: exclude the last 3 days from deleting session files
UserDownloadQueueForm: fix bug
2024-03-22 01:28:49 +03:00
Andy
2a2c12c651 2024.3.19.0
API.XHamster: some videos are missing when downloading creators; user videos aren't downloading
2024-03-19 01:09:54 +03:00
Andy
2bacc17ac4 2024.3.17.0
API.OnlyFans: add stories download
API.PornHub: fix list trim
2024-03-18 02:27:01 +03:00
Andy
7a68067d77 2024.3.15.0
PluginProvider
Add 'Checked' property
Add 'PropertyValueEventArgs'

SCrawler
Add default headers
Add predefined colors to the color picker
API.Instagram: automatically reset download options after updating credentials
2024-03-15 21:38:55 +03:00
Andy
d1eacc2db2 2024.3.13.0
YT
Add the ability to add downloaded item(s) to a playlist
Add the ability to create a playlist from downloaded music

SCrawler
Fix resetting of 'Available' indexes when the exiting a job
Allow thumbnail to be saved with file
API.TikTok: add more settings for standalone downloader; fix files with long names aren't downloaded (STD)
Feed: add moved/copied files to message box
2024-03-13 23:34:00 +03:00
Andy
3f38803643 2024.3.4.0
YT
Add the ability to add downloaded item(s) to a playlist
Add the ability to create a playlist from downloaded music

SCrawler
Fix resetting of 'Available' indexes when the exiting a job
2024-03-04 21:09:38 +03:00
Andy
09760a6926 2024.2.25.0 2024-02-25 11:31:33 +03:00
Andy
75039ac4d2 2024.2.24.0
Feed: improve move/copy
2024-02-24 12:24:05 +03:00
Andy
03e3a07947 2024.2.22.1
Feed: add the ability to update file location when moving media
Automation: make the automation file relative
2024-02-22 19:58:46 +03:00
Andy
d283d9c9f9 2024.2.22.0
Feed: add the ability to move/copy media
2024-02-22 13:40:04 +03:00
Andy
b9100bd3c1 2024.2.21.0
API.UserDataBase: incorrect comparison when 'Other' is 'UserDataBind'
DownloadFeedForm: add saved session name to feed title; remove menu hiding when session saving is disabled; remove menu tooltip
GlobalSettingsForm: update 'FeedParametersChanged'
2024-02-21 09:19:40 +03:00
Andy
0ea2156ada 2024.2.17.0
YT
Add the ability to edit playlist items
Add 'Open file' to the context menu
Add the ability to embed thumbnail in the audio/video as cover art
VideoOptionsForm: audio codec does not change when changing audio/video in the video options form

SCrawler
DownloadFeedForm: add ability to merge multiple special feeds into one
AutoDownloader: fix bug when users are added during pool reconfiguration
Scheduler: add the ability to move tasks
FeedMedia: fix image rendering bug
Feed: add select all/none; add the ability to add to a special feed(s) with removal from the current one; add loaded feed name to the title; refresh the loaded feed using the 'Refresh' button
FeedSpecialCollection: add 'Add' button to feed chooser; fixed a bug in the 'Delete' function
SettingsHostCollection, PluginHost: add 'IDisposable' support
API.UserDataBase: add Responser handler options
API.OnlyFans: handle 500 error
API.Threads: extract 'csrftoken' from cookies; simplify 500 error when updating tokens
API.Instagram: update handling of JSON parsing error when downloading reels; fix error downloading single post
API.Facebook: simplify token update errors
API.Twitter: update handling of JSON parsing error
2024-02-18 00:10:39 +03:00
Andy
520280b038 2024.2.12.0
API.Instagram: extract 'csrftoken' from cookies; remove 'x-ig-www-claim' from settings
2024-02-12 10:28:21 +03:00
Andy
10516f229b 2024.2.10.0
Plugins: added `ReplaceInternalPluginAttribute` attribute

SCrawler
API: update user regex for some sites
API.Instagram: simplify 5xx errors; hide JSON deserialization error
API.TikTok: files with long names aren't downloaded (PathTooLongException)
2024-02-10 08:19:08 +03:00
Andy
c3f0831768 2024.2.9.0
Add a 'Feed' button to notifications
SCrawler.StandaloneDownloader: url array form doesn't show scrollbars
2024-02-09 10:00:04 +03:00
Andy
74a0404670 Update README.md 2024-02-03 08:19:53 +03:00
Andy
52a43b9207 2024.1.26.0
YT
YouTubeSettings: add property DefaultVideoFPS
VideoOptionsForm, YouTubeMediaContainerBase: add FPS reduction

SCrawler
API.Instagram: change back aspect ratio determining
API.TikTok: add the ability to use a regex to clean the title
API.YouTube: add the ability to ignore community errors
2024-01-26 04:13:42 +03:00
Andy
5bc559c448 2024.1.20.0
API.Instagram: add reels support (separate)
API.LPSG: handle 404 error
2024-01-20 00:00:23 +03:00
Andy
b37f641582 2024.1.18.0
YT: url array form doesn't show scrollbars
API.Instagram: change aspect ratio determining
API.xHamster: some user videos were not downloaded
API.UserDataBind: incorrect collection sorting
DownloadFeedForm: change separator in result dialog when merging feeds
2024-01-18 01:23:55 +03:00
Andy
e00dfec701 2024.1.12.1
API.Instagram: stories (user) downloading with the wrong aspect ratio for some users
API.YouTube: fix incorrect opening of a post from the feed; fix wrong date to data parsing; add data downloading by dates

DownloadFeedForm: add merging multiple session feeds into one
2024-01-12 19:58:17 +03:00
Andy
94edf23570 2024.1.12.0
DownloadFeedForm: add an option to create a new feed when adding checked items; add a prompt before clearing the current session
MainFrame: add scheduler to tray menu

API.Instagram: fix tagged posts downloading
API.xHamster: fix profiles downloading; add creators downloading
API.YouTube: add error to log (communities)
2024-01-11 23:39:56 +03:00
Andy
0246af9b69 2023.12.27.0
API.OnlyFans: add OF-Scraper support
ProfileSaved: save files when adding new data
2023-12-27 16:10:02 +03:00
Andy
c458f1cd1d 2023.12.26.0
SiteEditorForm: sort controls only if some of them have numbers
UserCreatorForm: reset site options for new user only; add host reset when changing account for created user
PropertyValueHost: add 'UpdateMember' function
SettingsHost: add 'DefaultInstanceChanged' function to update properties when setting default instance
SettingsHostCollection: update properties when setting default instance
2023-12-26 14:34:53 +03:00
Andy
e3da1bf1d3 2023.12.21.0
Update 'new log data' notification
Feed: improve last session loading
2023-12-21 10:01:30 +03:00
Andy
dde024276b 2023.12.20.0
API.Twitter: simplify JSON error string
Add notification of new log data
Add deleting permanent cache path if empty
2023-12-20 15:37:40 +03:00
Andy
7a33c02497 2023.12.15.0
Fix 'GetCurrentMaxVer' bug
2023-12-14 23:51:35 +03:00
Andy
5bd79ff3c2 2023.12.14.1
API.Twitter: add additional nodes
2023-12-14 23:06:55 +03:00
Andy
3268bca0d3 2023.12.14.0
YT
YouTubeSettings: add 'CreateThumbnails_Video' and 'CreateThumbnails_Music' properties
VideoListForm: add 'BTT_SELECT_ALL' and 'BTT_SELECT_NONE'
YouTubeMediaContainerBase: update thumbnails downloading
2023-12-14 14:19:20 +03:00
Andy
1455a31879 2023.12.13.0
YT
Structures: add 'YouTubeChannelTab' enum
YouTubeFunctions: add 'StandardizeURL_Channel' function; update 'Info_GetUrlType', 'Parse' and 'Parse_Internal' functions (YouTubeChannelTab)
YouTubeSettings: add 'ChannelsDownload' property and 'YouTubeChannelTabConverter' converter for grid
Add 'ChannelTabsChooserForm'
VideoListForm: remove buttons 'BTT_ADD_NO_SHORTS' and 'BTT_ADD_SHORTS_ONLY'; update 'BTT_ADD_KeyClick' function

SCrawler
API.M3U8Base: add 'M3U8URL' structure; update 'Download' function with 'OnlyDownload' arg
API.ProfileSaved: add sorting of feed files
API.Xhamster: update M3U8 parser and 'Download' function for additional downloading ways; add 'ReencodeVideos' property (to the 'SiteSettings')
API.YouTube: update to updated 'Parse' function
2023-12-13 19:48:50 +03:00
Andy
64d6e6b28c 2023.12.10.0
YT: move updater functions into the app
SCrawler.API.Twitter: update parsing function to new GDL (1.26.4-dev)
2023-12-10 10:16:58 +03:00
Andy
da7cddc720 Update README.md 2023-12-07 14:08:32 +03:00
Andy
72be6b09ff Update README.md 2023-12-07 11:33:52 +03:00
Andy
3a0837a1b0 2023.12.7.0 2023-12-07 11:03:34 +03:00
Andy
0657f3d195 2023.12.6.1
Update updater
YT: add new version check at start
2023-12-06 18:28:36 +03:00
Andy
c92314d8e8 2023.12.6.0
Add updater
2023-12-06 10:55:50 +03:00
Andy
d99243ce46 2023.12.5.0
YT
VideoListForm: add a check of adding a URL if it has already been downloaded ('ValidateContainerURL')
YouTubeMediaContainerBase: add 'GetUrls' and 'GetFiles' functions; make 'Files' protected friend; update 'CreateUrlFile' function

SCrawler
Add downloaded saved posts to the feed
API.ProfileSaved: add token verification for multi-acc
API.SiteSettingsBase: update 'UpdateResponserFile' and 'CLONE_PROPERTIES.filterUC' functions
API.UserDataBase: add a null host check before request a new key; update 'OpenFolder' function (for saved posts)
API.YouTube: add the ability to download YouTube user community feeds
DownloadProgress: add 'KeyClickEventArgs' to download saved posts excluding from feed; add 'FeedFilesChanged' event; update 'Start' function
DownloadSavedPostsForm: add 'FeedFilesChanged' event and handler; update 'Start' function
Feed.FeedMedia: make the class compatible to work with saved posts
StandaloneDownloader.VideoDownloaderForm: add a check of adding a URL if it has already been downloaded ('ValidateContainerURL')
TDownloader: add the 'IsSavedPosts' field to the 'UserMediaD' structure; update 'UserMediaD.New(EContainer)' function (for saved posts); update 'UserMediaD.ToEContainer' function; add 'SessionSavedPosts' property
MainFrame: add 'Alt+A' hotkey to show scheduler; add 'Alt+P' hotkey to show progress form
Hosts.DownloadableMediaHost: add URL file to files list
2023-12-05 12:05:20 +03:00
Andy
3dae40b696 Update Plugins.md 2023-12-04 07:30:21 +03:00
Andy
ee0c773c37 Update README.md 2023-11-26 03:18:04 +03:00
Andy
ebe5f0ca01 2023.11.25.0
SettingsHost, SiteSettingsBase: change comparer to 'MembersDistinctComparerExtended'
API.RedGifs: fix initial UserAgent value
AutoDownloader: add 'IsManual' to 'Copy' function
2023-11-25 03:00:14 +03:00
Andy
a540aded68 2023.11.24.0
Scheduler: handle scheduler change error (collection was modified)
SchedulerEditorForm: add scheduler name to form title
DownloadProgress: fix disposing error when some objects are already null
API.Reddit: add special notification for error 429
API.Twitter: handle JSON deserialization error
Porn sites: fix incorrect parsing of search queries
YouTube: path not set when adding array to download
2023-11-24 04:54:04 +03:00
Andy
0ec617c1dc 2023.11.21.0
YouTube
Add absolute paths support
PlaylistArrayForm: fix RTB issue
Single media: the file name is not changed manually

SCrawler
Automation: add manual tasks
DownloaderUrlsArrForm: fix RTB issue
SiteSettingsBase: add use of Netscape cookies if enabled for a class; disable saving Netscape cookies on init
Add feeds update when users' location and/or basic info changes
API.TikTok: add ID, username and friendly name extraction from data; update request URL; update 'GetUserUrl' function; add new option 'Use video date as file date'
API.YouTube: set 'UseNetscapeCookies'
2023-11-21 09:25:22 +03:00
Andy
45adf735a7 2023.11.17.0 2023-11-17 02:30:23 +03:00
Andy
496c9487cd 2023.11.15.0
ADD FACEBOOK
SiteSettingsBase: update 'CLONE_PROPERTIES' function (exclude 'DoNotUse' attribute)
API.Instagram: handle 401 error
API.ThreadsNet.SiteSettings: make the class compatible for Facebook
xHanster, XVideos, PornHub, ThisVid: update download function for search queries
Hosts.PropertyValueHost: set the 'Exists' value based on the 'DoNotUse' attribute
Hosts.SettingsHost: use 'GetObjectMembers' instead of 'GetTypeInfo.DeclaredMembers' to get class members
2023-11-17 02:30:23 +03:00
Andy
96705f1c59 2023.11.13.1
PornHub: add playlists downloading
2023-11-13 23:18:21 +03:00
Andy
f08a5f9259 2023.11.13.0
API.JFF: update UserAgent start value
ActiveDownloadingProgress: fix unnecessary focus of inactive form
FeedMedia: add additional buttons to the context menu
FeedSpecialCollection: 'Favorite' feed does not appear in the feeds list if it is created manually
UserCreatorForm: disable account selection when opening an empty form
MainFrame: fix background image resizing on form state changes
SettingsHost: inherit path when cloning an instance
SettingsHostCollection: Extract user path change function to static
2023-11-13 20:53:32 +03:00
Andy
95cbb6aeb1 2023.11.12.2
API.Instagram: handle JSON primitive error (simple line)
API.OnlyFans: handle 401 error; handle file update error
API.Xhamster: handle 503 error
Scheduler: handle automation start error
2023-11-12 22:34:47 +03:00
Andy
5af0dcc46e 2023.11.12.1
YouTube: add cc and subtitles merge
2023-11-12 20:57:05 +03:00
Andy
f5789862ba 2023.11.12.0
Add special feeds
2023-11-12 20:34:53 +03:00
Andy
12c02580f6 2023.11.9.0
ADD MULTI-ACCOUNT

PluginProvider
IDownloadableMedia: added 'AccountName' property
IPluginContentProvider: added 'AccountName' property
ISiteSettings: added properties: 'AccountName', 'Temporary', 'AvailableText', 'DefaultInstance'; added functions: 'Clone', 'Update', 'Delete'; removed 'Load' function; implement 'IDisposable' interface
PropertyValue: added functions: 'BeginInit', 'EndInit', 'Clone'

YT
YouTubeSettings: make the class compatible for multi-acc
YouTubeMediaContainerBase: add 'AccountName' property

SCrawler
IUserData: add properties: 'HostStatic', 'AccountName'
ProfileSaved: add the ability to download saved posts from all accounts
SiteSettingsBase: add multi-acc support; add 'UserOptionsType' for future purposes; update initializers; update responser initializing; add 'CLONE_PROPERTIES' and 'CloneGetEmptySettingsInstance' functions; 'IDisposable' support
UserDataBase: add multi-acc support; change host retrieval method
DomainsContainer: implements 'IDisposable' interface

API.All sites: add multi-acc support; move the Icon and Image setting to the initializer; update initializer
API.Instagram: change some property types
API.Reddit: set 'AvailableText'; update 'UpdateRedGifsToken' and 'UserOptions' functions
API.Mastodon: remove 'MastodonDomains' class and 'SettingsForm' form; replace 'MastodonDomains' with 'DomainsContainer'; update functions 'IsMyUser' and 'IsMyImageVideo'; update to 'DefaultInstance' environment; update 'UserData.ResetCredentials' function
API.XVIDEOS: update to 'DefaultInstance' environment
API.Xhamster: update to 'DefaultInstance' environment

STDownloader: add multi-acc compatibility

SiteEditorForm: add option 'Download saved posts'; update providers; add additional providers; add multi-acc support

PluginsEnvironment: add 'PClonableAttribute'; add multi-acc support
2023-11-12 01:31:09 +03:00
Andy
6def34d5e9 2023.10.12.0
API.ThisVid: update parser ('SessionPosts')
2023-10-12 19:47:02 +03:00
Andy
831d9a5a09 2023.10.10.0 2023-10-10 18:10:45 +03:00
Andy
326e61a968 2023.10.9.0
YT.VideoListForm: hide clear and delete buttons in menu; add 'BTT_CLEAR_SELECTED' button
API.Base.TokenBatch: add debug option
API.ALL: fix missing posts
API.JFF: rewrite m3u8 downloader; add ffmpeg requirement for the download; fixed missing posts; fixed download to the date; fixed corrupted files
DownloadableMediaHost: remove thumbnail when removed from list if thumbnail is stored in cache
2023-10-09 18:52:37 +03:00
Andy
61903afe3f 2023.10.6.0
Collections: use the collection name as a friendly name; labels are removed when creating a new collection
API.UserDataBase: update comparator
API.ThisVid: add 'PagePosts' to continue parsing when new posts are added
2023-10-06 15:11:09 +03:00
Andy
fa3f39b905 2023.10.4.0
YT: file name is missing when destination is changed by selecting one of the saved locations; missing files still appear in the list
API.UserDataBase: hide 'OperationCanceledException' and 'ObjectDisposedException'
API.Reddit: unable to save settings without OAuth data
TDownloader: 'ReconfPool'
MainFrame: replace 'DownloadQueue.FormShow' with 'ShowDownloadQueueForm'
SettingsHost: add a notification if the user has disabled downloading from the site
2023-10-04 22:53:17 +03:00
Andy
c76fd7f918 2023.10.3.0
YT: add settings 'CreateUrlFiles' and 'CreateDescriptionFiles'
STDownloader: add setting 'CreateUrlFiles'
2023-10-03 23:33:42 +03:00
Andy
66085e0d95 2023.10.2.0
API.Threads: save new token if found
Feed: add 'Load current session'; update the session file retrieval function
TDownloader: change session files name; files count
SettingsCLS, GlobalSettingsForm: add session file count setting
2023-10-02 11:32:37 +03:00
Andy
0a1602b453 2023.10.1.0
JustForFans: some profiles won't download
2023-10-01 18:35:08 +03:00
Andy
adf788781d 2023.9.30.1
YouTube: add URL standardization; change the 'Browse' button handler in 'VideoOptionsForm'
API.UserDataBase: add 'TokenQueue' to main function exceptions
2023-10-01 18:02:53 +03:00
Andy
77711965c0 2023.9.30.0
Add Threads.net

API.UserDataBase: add 'EraseData_AdditionalDataFiles' function; add 'ThrowAnyImpl' function; add 'e' arg to 'LogError' function
API.Instagram: make classes compatible with threads.net; add top limits; update container parsers; add an override to the 'EraseData_AdditionalDataFiles' function
2023-09-30 09:16:57 +03:00
Andy
7f1ac6f512 2023.9.29.0
UserDataBind: fix labels, colors, script when adding a new user
UserCreatorForm: fix labels issue
2023-09-29 16:24:43 +03:00
Andy
f4eb33d8da 2023.9.28.0
API.Mastodon: hide 503 error
API.PornHub: minor fixes
API.RedGifs: fix 'DataGone'
Minor bugs
2023-09-28 17:38:34 +03:00
Andy
77443cedc4 2023.9.21.0
PornHub: videos are not downloading
2023-09-21 06:22:17 +03:00
Andy
a446df1f66 2023.9.20.0 2023-09-20 12:17:00 +03:00
Andy
0026e905a4 2023.9.19.0
YT: add priority download protocol
2023-09-19 12:55:53 +03:00
Andy
f8116fd048 2023.9.18.0
API.Instagram: handle error 500; fix saved posts bug
API.Reddit: disable token refresh if there are no profiles to download
API.UserDataBind: consolidate colors; update labels only for the added user
AutoDownloader: change pause event; update pause function for scheduler; fix incorrect pause in scheduler; add icon for SchedulerEditorForm
2023-09-18 07:54:26 +03:00
Andy
8d33fdc8f3 Update README.md 2023-09-18 07:34:00 +03:00
Andy
dab94acc32 2023.9.6.0
Add scheduler changer
2023-09-06 23:38:16 +03:00
Andy
c61c817585 2023.9.3.0
API.Instagram: add user (non-pinned) stories
API.UserDataBase: fix 'StartMD5Checked' initial value
2023-09-03 19:54:01 +03:00
Andy
3ea59a6acd 2023.8.27.0
YT: fix 'Shorts' downloading
2023-09-03 19:38:18 +03:00
Andy
2a60ace18f 2023.8.27.0
API.JFF: remove PXML attribute for some properties
API.Reddit.Channels: save channel info right after download; replace date providers with default
API.Reddit.SiteSettings: improve 'UpdateToken' function
AutoDownloader: add 'Copy' function
SchedulerEditorForm: add cloning plans
DownloadedInfoForm: add 'Try...Catch' for some functions
DownloadFeedForm: add button to go to custom page
FeedMedia: color typo
GroupParameters: add 'ICopier'
2023-08-27 19:39:50 +03:00
Andy
f0014d2874 2023.08.19.0
API.Base: update 'TokenBatch', 'GDLBatch'; add 'YTDLPBatch'
API.Base.UserDataBase: update 'EraseData' function
API.Redgifs: fix hd/sd issue
API.TikTok: update dates in commands; replace 'TokenBatch' with 'YTDLPBatch'
UserDownloadQueueForm: update functions
2023-08-19 18:42:01 +03:00
Andy
28ae44f0ae 2023.08.17.0
YT.VideoListForm: hide progress

API.Base.UserDataBase: add 'IsUser' property; remove 'DownloadedPictures' debug line; add a special log for non-existent users
API.Twitter: group 'limit' notifications; update 'TwitterLimitException' (inherits Plugin.ExitException)
AutoDownloader: fix 'Initialization' value bug
DownloadedInfoForm: fix a bug due to which profiles were disposed
FeedMedia: add subscriptions users BackColor & ForeColor; fix file name issue; remove icon cloning
TDownloader, UserDownloadQueueForm: fix progress hang issue
ColorPicker: add 'TooltipText'
GlobalSettingsForm: add new properties; move design properties to new tab
ListImagesLoader: add subscriptions users BackColor & ForeColor
2023-08-17 23:54:19 +03:00
Andy
1b1226025a 2023.08.10.0
Add JFF

Update groups
Add advanced filter
Add advanced download
Disable 'ShowInTaskbar' on several forms
API.Base.M3U8: add external cache support
API.Base.UserDataBase: update token names
Feed.FeedMedia: add clone icon
UserCreatorForm: fix bug collection labels not showing
MainFrame: update 'DownloadSiteFull' function
UserSearchForm: move focus to textbox on form is open
2023-08-10 22:42:29 +03:00
Andy
58927b3113 Update README.md 2023-08-08 17:17:23 +03:00
Andy
df06a86651 2023.8.6.0
Plugins.Attributes: add 'DependentFields' attribute
Plugins.IPluginContentProvider: add 'Options' and 'IsSubscription' properties
Plugins.ISiteSettings: add 'SubscriptionsAllowed' property
Plugins.ExchangeOptions: add 'Options' field
Plugins.Attributes.PropertyUpdater: replace 'Dependencies' with 'Arguments'

YT: add 'OutputPathAskForName' and 'OutputPathAutoAddPaths' properties; add the ability to store download locations; add 'DownloadLocation' and 'DownloadLocationsCollection' objects
YT.IDownloaderSettings: add 'OutputPathAskForName' and 'OutputPathAutoAddPaths' properties
YT.Downloader: fixed bug with re-saving elements when loading a video list; fixed bug when files were not deleted when clicking on the delete button; fixed a bug that caused the video to redownload; download job removes elements at wrong indexes; added skipping of downloaded elements in the job; fixed a bug, pending option did not change after download complete
YT.YouTubeMediaContainerBase: add '_MediaStateOnLoad' field and 'NeedToSave' function; update the 'Save' function to prevent saving a file when a download is complete and the file has already been saved; update code for new yt-dlp version

Fixed cache deletion errors
Add user queue
Add global locations
API.Base.SiteSettingsBase: implement 'SubscriptionsAllowed' property; remove request headers with null values on save; add '_AllowUserAgentUpdate' parameter
API.Base.Structures: add 'SiteModes' enum
API.Base.UserDataBase: add 'Erase' button; implement 'Options' and 'IsSubscription' properties; add 'SpecialLabels' property; update 'LVIKey'; update 'FitToAddParams' function; add 'EraseData' function; user colors; Not UserExists notification, UserQueue support
API.Base: add 'DeclaredNames'
API.Instagram: remove default values for headers; disable updating UserAgent from global; check for a new username for non-existent users
API.Mastodon: bypass new inherited twitter options; update names and headers
API.OnlyFans: make 'HH_BROWSER' property nullable; remove 'HH_BROWSER' from required; fix username bug (dots); handling of 504 and 429 errors; add 'DownloadHighlights' and 'DownloadChatMedia' options; add 'UserExchangeOptions'; fixed incorrect error handler
API.PathPlugin: fixed incorrect detection of path existence
API.Pinterest: add 'SpecialLabels'
API.PornHub: add new video regex; remove old regex; added 'DownloadUploaded', 'DownloadTagged', 'DownloadPrivate' and 'DownloadFavorite' properties to 'SiteSettings', 'UserData' and 'UserExchangeOptions'; update regex to define user; added downloading search queries; update 'GetUserUrl' function; hide unnecessary 'RegexFieldsTextBecameNullException' errors; add subscriptions
API.Reddit: add 'SpecialLabels'; add bearer token and its refresh interval; add OAuth; add additional options
API.RedGifs: add 'DependentFields' for 'Token'
API.ThisVid: add 'DownloadFavourite' option; add downloading search queries, tags, categories; add 'SpecialLabels'; add subscriptions; updating cookies issue
API.TikTok: rewrite algorithms
API.Twitter: add 'UseAppropriateModel', 'UseNewEndPointSearch', 'UseNewEndPointProfiles', 'AbortOnLimit', 'DownloadAlreadyParsed', 'MediaModelAllowNonUserTweets' properties; remove old commented code; remove 'TwitterPic_400' and replace with 'TwitterIcon_32.ToBitmap'; add 'DownloadModelForceApply' user option; update environment to GDL 1.25.8; fixed gifs downloading; fix typo in 'ReparseMissing'; update names
API.UserDataBind: prevent adding site-specific labels when adding to a collection
API.Xhamster: add downloading search queries, tags, categories; add 'SpecialLabels'; add additional nodes for channels; add subscriptions
API.XVIDEOS: add downloading search queries, tags, categories; add 'SpecialLabels'; add subscriptions; changed users creation method; add subscriptions
API.YouTube: add subscriptions
AutoDownloader: add new group subscription options; update predicates; fixed excluded labels and sites in default mode; update notifications; add an additional skip options, add 'Force start' option
DownloadedInfoForm: add subscriptions; fixed size/location bug; hide unnecessary error (refill)
Feed: add subscriptions; update filters; add 'Ctrl+G' shortcut
FeedMedia: add subscriptions; fixed 'webm' bug; add title for subscription media; add site icon to post; user colors; always using 'FriendlyName' instead of 'UserName' if it exists
DownloadGroup, GroupDefaults, GroupParameters: add subscription and 'UsersCount' options
MissingPostsForm: add 'BTT_DELETE_ALL'
VideoDownloaderForm, DownloaderUrlForm, DownloaderUrlsArrForm: add download locations support
VideoDownloaderForm: add subscriptions support
GlobalSettingsForm: add new properties
UserCreatorForm: add subscriptions; add 'Options' support (of 'ExchangeOptions'); user colors
ListImagesLoader: add subscription colors; user colors
MainFrame: add subscriptions; add filters by subscription and user; update predicates
NuGet: update 'LibVLCSharp', 'LibVLCSharp.WinForms', 'VideoLAN.LibVLC.Windows'
DownloadableMediaHost: update 'Save' function
PropertyValueHost: fix 'CaptionWidth' bug; add 'Dependents'
SettingsHost: add 'Dependents'
UserDataHost: add 'Options' and 'IsSubscription' properties
SettingsCLS: implement new 'IDownloaderSettings' properties; add 'CacheSnapshots'; add 'DownloadLocations'; add new properties
UserInfo, UserFinder: add subscriptions
UserSearchForm: fixed search by name bug
2023-08-06 18:16:07 +03:00
Andy
bade8666d5 Update README.md 2023-06-29 20:54:27 +03:00
Andy
c70caa0035 Update README.md 2023-06-23 09:40:40 +03:00
Andy
ac532dbc6f Update README.md 2023-06-21 14:00:25 +03:00
Andy
82ef4f4410 2023.6.19.0
YT.Progress: make the playlists parsing progress more informative; change form display method
YT.YouTubeMediaContainerBase: fix sort algo
YT.Tray: add 'Add' button; add 'Ctrl+Click' on tray icon to add download
YT.Settings: add setting 'Download on click in tray: show form'
LPSG: some files didn't download (encoding)
Twitter: hide cache deletion errors
Mastogon: fixed bug in 'ReparseMissing' function
Reddit: downloaded gifs are static
XHamster: videos are not downloading or downloading incorrectly
Progress: fix bugs; minor improvements
2023-06-19 06:05:28 +03:00
Andy
d34414359c 2023.6.9.0
YT.MediaItem: fixed opening paths to downloaded playlists and channels
API.InternalSettingsForm: add members distinct
API.Mastodon: create personal EditorExchangeOptions class with some Twitter members disabled
API.Twitter: add 'DownloadModels'; update algo
Make progress more informative
2023-06-09 21:44:00 +03:00
Andy
e51debc027 2023.6.8.0
YT.Music: append artist name to music playlist output path
YT.MediaItem: fixed opening paths to downloaded playlists and channels
YT.YouTubeMediaContainerBase: save thumbnail path for playlist and channel
UserDataBase: remove old line of code
API.Twitter: fixed profile not fully downloaded
SiteEditorForm: corrected form size for small monitors
2023-06-08 17:27:19 +03:00
Andy
938042ea9e 2023.6.5.0
YT settings: removed property 'ItemsListLimit', add property 'ReplaceModificationDate'
YT.MediaItem: fix 'Pending'
YT.VideoListForm: add 'Shift' to add without downloading; add 'F5' hot key to start download; remove list items limit; fix item 'Pending', fixed items queue

UserDataBase: add 'IconBannerDownloaded' properties; add 'HOST.Available' check to 'DownloadSingleObject'; update file deletion in 'DownloadContentDefault'; add truncating '_TempPostsList' if number of ids > 1000
Instagram: add authorization headers
Mastodon: implement 'DownloadIconBanner'; update 'ReparseMissing' function
Reddit: implement 'DownloadIconBanner'
Twitter: implement 'DownloadIconBanner'; update parsers to parse posts with two videos; implement gallery-dl for all function; remove headers from settings
Download.DownloadProgress: remove main progress perform when downloading saved posts
VideoDownloaderForm: bind the 'BTT_ADD_URLS_ARR' button to the 'BTT_ADD_KeyClick' function
UsersInfoForm: add folder opening on double click on an item
ListImagesLoader: fix refill bug when the number of filtered profiles = 0
TrayIcon: add standalone downloader to context menu
DownloadableMediaHost: fix a bug when not downloaded videos do not appear in the list when loading the program
2023-06-05 19:36:35 +03:00
Andy
abdef81e5f Update README.md 2023-06-03 11:51:01 +03:00
Andy
e868c2e694 2023.5.12.0
IPluginContentProvider: add 'ProgressPreChanged' and 'ProgressPreMaximumChanged' events
YT.MediaItem: change folder opening on double click
YT.VideoListForm: change the icon for the 'Download' button

Add advanced progress
Add user metrics calculation
UserDataBase: fix GIF hash bug
Instagram: heic to jpg
Mastodon.SiteSettings: add the main domain to the list of domains with saving the settings
Mastodon.UserData: handle 'Forbidden' error; fix bug in parsing non-user posts
Pinterest: remove cookies requirement for saved posts
PornHub: fix resolutions issue; add 'DownloadUHD' option
Reddit: fix missing images bug; fix broken images bug; update container parsing function
MainFrame: fix collection pointing bug
2023-05-12 20:00:32 +03:00
398 changed files with 44256 additions and 7186 deletions

View File

@@ -30,7 +30,9 @@ A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Release information (please complete the following information):**
**Release information:**
**Please complete the following information or replace the following text with data copied from SCrawler (click the top right info button in the main window, then the `Environment` button, then the `Copy` button, and paste the copied text here).**
- OS: [e.g. Windows 10, Windows 11]
- Architecture: [e.g. x86, x64]
- Version: [e.g. 2023.3.5.0]
@@ -38,6 +40,7 @@ If applicable, add screenshots to help explain your problem.
- ffmpeg version (command `ffmpeg -version`):
- yt-dlp version (command `yt-dlp --version`):
- gallery-dl version (command `gallery-dl --version`):
- cURL version (command `curl --version`):
**Additional context**
Add any other context about the problem here.

3
.gitignore vendored
View File

@@ -10,8 +10,7 @@
*.userosscache
*.sln.docstates
.obsidian/
ToDo.txt
ToDo.md
BugReporterFormDiscordWebHook.vb
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs

View File

@@ -1,11 +1,14 @@
# Contributor's Guide
I welcome requests! Follow these steps to contribute:
Follow these steps to contribute:
1. Find an [issue](https://github.com/AAndyProgram/SCrawler/issues) that needs assistance.
1. Let me know you are working on it by posting a comment on the issue.
1. If you find an error in the code, please provide a link to the file and the line number.
1. If you have a code change suggestion, you can post a replacement code block. I also accept pull requests.
1. Let me know you're working on this by posting a comment on this issue.
1. If you find a bug in the code, please provide a link to the file and line number.
1. If you have a code change suggestion, you can post a replacement code block.<!-- I also accept pull requests.-->
# How to report a problem
**[Read here](https://github.com/AAndyProgram/SCrawler/blob/main/FAQ.md#how-to-report-a-problem)**
# How to build from source
1. Delete the `PersonalUtilities` project from the solution.
@@ -24,7 +27,7 @@ I welcome requests! Follow these steps to contribute:
**I'm currently not accepting requests to develop new sites.**
1. Check [issues](https://github.com/AAndyProgram/SCrawler/issues) (open and [closed](https://github.com/AAndyProgram/SCrawler/issues?q=is%3Aissue+is%3Aclosed)) and [discussions](https://github.com/AAndyProgram/SCrawler/discussions) to find your issue. Perhaps I have already answered your request.
1. If you don't find anything, create a new issue with your request. I usually reply as soon as possible (within the next few hours).
1. If you don't find anything, create a new issue with your request.
# Requirements for new site requests
@@ -39,5 +42,4 @@ I welcome requests! Follow these steps to contribute:
If I'm interested in a site you want to add, it may be added in future releases.
# Sites I will never develop
- Facebook
- Tumblr

View File

@@ -1,3 +1,923 @@
# 2025.6.12.0
*2025-06-12*
- Updated
- yt-dlp up to version **2025.06.09**
- Fixed
- Sites:
- YouTube: audio formats of protocol `m3u8` are not handled correctly
- BlueSky: data is not downloaded in some cases
- Reddit: new users do not inherit default text settings
- Saved posts: text downloading with saved posts
- Environment incorrect output
# 2025.6.1.0
*2025-06-01*
- Added
- Sites:
- YouTube (standalone app):
- support for **youtu.be** domain
- **filters**
- hotkey `Ctrl+F5` for refresh
- **Bluesky, Instagram, OnlyFans, Reddit, Threads, Twitter: the ability to download text posts**
- OnlyFans:
- updated `DynamicRules`
- `backend` option *(`aio` & `httpx`)*
- Reddit: add `Best` and `Rising` view modes
- TikTok: downloading of avatar, site name and description
- Feed:
- **filters**
- add the ability to show test posts
- Minor improvements
- Updated
- yt-dlp up to version **2025.05.22**
- gallery-dl up to version **1.29.7**
- PluginProvider
- IUserMedia: properties `PostText`, `PostTextFile`, `PostTextFileSpecialFolder`
- Fixed
- Sites:
- OnlyFans:
- DRM videos are not downloading in some cases *(only if you can't download video try changing `backend` option to `httpx`)*
- incorrect open posts
- while limited downloading, the first time, the profile still loads completely
- Reddit:
- **data is not downloading**
- post date is incorrect
- Threads
- change domain from `net` to `com`
- data is not downloading
- Twitter
- fix JSON error
- data is not downloaded in some cases
- Minor bugs
# 2025.3.17.0
*2025-03-17*
- Added
- **TikTok: downloading photos**
- Updated
- gallery-dl up to version **1.29.2**
- Fixed
- Sites
- Facebook: reels aren't downloaded from noname profiles
- PornHub: newly added users aren't downloading
- Threads: users aren't updated if there is a pinned post
# 2025.2.25.0
*2025-02-25*
- Added
- Sites:
- **Bluesky**
- Facebook: **`Reels` downloads**
- OnlyFans: default value for `App-Token`
- Pinterest: **sub-boards downloading**
- Threads: ability to manually change `UserName`
- Twitter:
- new icon support
- **sleep timers to fully download large profiles**
- Feed:
- ability to invert selection
- open post URL when double-clicking on subscription image
- Minor improvements
- Updated
- yt-dlp up to version **2025.02.19**
- gallery-dl up to version **1.28.5**
- PluginProvider
- `IPluginContentProvider`: added property `NameTrue`
- Fixed
- Sites:
- Facebook: videos are not downloading
- LPSG: simplified 403 error
- PornHub: photos & videos are not downloading
- Reddit: **token does not update automatically**
- Threads: **data is not downloading**
- Minor bugs
# 2025.1.12.0
*2025-01-12*
- Added
- Sites:
- YouTube (standalone app):
- ability to add channel name to file name (`Add channel to file name`)
- adding channel name and video URL to info file
- OnlyFans: **built-in usage of DRM keys**
- Threads: automatically change `heic` extension to `jpg`
- Twitter: download broadcasts *(user option)*
- Minor improvements
- Updated
- yt-dlp up to version **2024.12.23**
- gallery-dl up to version **1.28.3**
- **OF-Scraper** up to version **3.12.9** *(you must update it personally)*
- Fixed
- Sites:
- DownDetector: fixed 403 error
- OnlyFans: **DRM videos not downloading**
- xHamster: some videos are not downloading
- YouTube: **communities are not downloading** *(see settings in wiki)*
- Minor bugs
# 2024.11.21.0
*2024-11-21*
- Added
- Sites:
- Instagram:
- setting to skip errors without disabling download *(site settings)*
- settings to force update of username and/or user information *(user settings)*
- setting to continue downloading profile if error 560 occurs while downloading user stories *(site settings)*
- improve username update algorithm
- YouTube: 404 error handling (community)
- Main window: add extra buttons for special download (limited and dated) in collection
- Global settings: ability to change the feed opening shortcut (`Ctrl+F`/`Alt+F` *(Settings - Behavior)*)
- Minor improvements
- Updated
- yt-dlp up to version **2024.11.18**
- gallery-dl up to version **1.27.7**
- Fixed
- Users: network paths aren't working
- Main window: in some cases users are not updated in the list
- Minor bugs
# 2024.10.24.0
*2024-10-24*
- Added
- YouTube (standalone app):
- settings `Embed thumbnail (cover)` and `Allow webm formats`
- changed cover selection for music downloads
- allow `webm` formats if there are no `mp4` formats via http protocol (issue #211)
- Sites:
- Instagram:
- **ability to manually change username**
- **mark user as non-existent if user `ID` cannot be obtained**
- Twitter: **ability to manually change username**
- Main window: add users search button to 'Info' menu
- Minor improvements
- Updated
- yt-dlp up to version **2024.10.22**
- gallery-dl up to version **1.27.6**
- Fixed
- YouTube (standalone app): adding incorrect playlist lines
- Reddit: incorrect UNIX date parsing
- Can't change data path (issue #206)
- Minor bugs
# 2024.9.2.0
*2024-09-02*
- Added
- Instagram: options to enable/disable image extraction from video
- Feed: **prompt before moving entire feed/session**
- Main window: hotkeys `Alt+U` and `Ctrl+U` to open the user search form
- Minor improvements
- Updated
- gallery-dl up to version **1.27.3**
- Fixed
- **OnlyFans**: data is not downloading
- YouTube (SCrawler): incorrect parsing of video page
- Minor bugs
# 2024.8.10.0
*2024-08-10*
- Added
- Feed: button to open file folder
- Updated
- yt-dlp up to version **2024.08.06**
- gallery-dl up to version **1.27.2**
- Fixed
- YouTube (standalone app): **video is being parsed using cookies but is not downloading** *(Issue #205)*
# 2024.8.1.0
*2024-08-01*
- Added
- Minor improvements
- Updated
- yt-dlp up to version **2024.08.01**
# 2024.7.24.0
*2024-07-24*
- Added
- YouTube (standalone app)
- ability to convert non-`AVC` codecs (eg `VP9`) to `AVC` (`Settings` - `Defaults Video` - `Convert non-AVC codecs to AVC`)
- add the ability to set the playlist creation mode: absolute links, relative links, or both (`Settings` - `Music` - `Create M3U8: creation mode`)
- Threads: **saved posts downloading**
- Feed
- hotkeys `Esc` and `Ctrl+W` to close the form
- the ability to search for missing files in *special feeds*
- Scheduler: the ability to execute a script after the scheduler plan is executed *(`Settings` - `Behavior`)*
- Main window:
- added hotkey `Ctrl+F` to show the feed
- changed the hotkey from `Ctrl+F` to `Alt+F` to show the search form
- Updated
- yt-dlp up to version **2024.07.16**
- Fixed
- YouTube (standalone app): video files with line breaks in the name do not download correctly
- OnlyFans: rules parsing bug
- Minor bugs
# 2024.6.25.0
*2024-06-25*
**ATTENTION! To support downloading of DRM protected videos (OnlyFans), please update OF-Scraper to version [3.10.7](https://github.com/datawhores/OF-Scraper/releases/tag/3.10.7) (download `zip`, not `exe`).**
- Added
- OnlyFans: **new dynamic rules updating algorithm**
- Feed: ability to set the last session as the current one
- Updated
- gallery-dl up to version **1.27.1**
- Fixed
- Minor bugs
# 2024.6.10.0
*2024-06-10*
- Added
- YouTube (standalone app): add option to add extracted MP3 to playlist (`Settings` - `Defaults Video` - `Add extracted MP3 to playlist`)
- Feed
- settings to show/hide site name and file type from media title
- ability to move/copy files of a loaded feed/session to another location
- ability to reset current session
- Fixed
- Minor bugs
# 2024.6.6.0
*2024-06-06*
**ATTENTION!**
1. **To support downloading of DRM protected videos (OnlyFans), please update OF-Scraper to version [3.10](https://github.com/datawhores/OF-Scraper/releases/tag/3.10) (download `zip`, not `exe`).**
2. **If there is a `OFScraperConfigPattern.json` file in the SCrawler settings folder, replace the text of the file with [this text](https://github.com/AAndyProgram/SCrawler/blob/main/SCrawler/API/OnlyFans/OFScraperConfigPattern.json).**
3. **Set the value to `Dynamic rules` (in the site settings) = `https://raw.githubusercontent.com/Growik/onlyfans-dynamic-rules/main/rules.json`.**
- Added
- OnlyFans: new OF-Scraper option (`keydb_api`)
- Minor improvements
- Fixed
- OnlyFans: **data is not downloading**
- Minor bugs
# 2024.6.4.0
*2024-06-04*
**If you were using the [`yt-dlp-TTUser`](https://github.com/bashonly/yt-dlp-TTUser) plugin, you should remove it because this plugin was added to yt-dlp itself! Read more [here](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-requirements).**
- Added
- Added highlighting of scheduler plans (working, stopped, pending, etc.)
- YouTube (standalone app): add option to add the video upload date before/after the file name (`Settings` - `Defaults` - `Add date to file name`)
- Twitter: **`Communities` downloading**
- Feed: ability to select one of the download sessions and set it as the current session
- Minor improvements
- Updated
- yt-dlp up to version **2024.05.27**
- gallery-dl up to version **1.27.0**
- Fixed
- Twitter: deleting user directory when redownloading missing posts
- Minor bugs
# 2024.5.19.0
*2024-05-19*
- Added
- YouTube (standalone app): add upload date to description (request #192) (`Settings` - `Info` - `Create description files: add upload date`, `Create description files: create without description`).
- Fixed
- YouTube (SCrawler): advanced settings are not saved when changed
# 2024.5.18.0
*2024-05-18*
- Added
- YouTube (standalone app): highlight frame rates higher/lower than this value (`Settings` - `Defaults Video` - `Highlight FPS (higher/lower)`).
- Sites
- Instagram: 'DownDetector' support to determine if the site is accessible
- Reddit: change the naming method of video files (hosted on Reddit) to the `YYYYMMDD_HHMMSS` pattern
- Twitter
- `Likes` downloading *(user settings)*
- **changed domain from twitter.com to x.com**
- Site settings: group options by category
- Minor improvements
- PluginProvider
- `PropertyOption` attribute: set category name when `IsAuth = True`
- `ISiteSettings`: added `UserAgentDefault` property
- Updated
- gallery-dl up to version **1.27.0-dev**
- Fixed
- Sites
- Instagram: incorrect definition of pinned posts
- Threads: new posts are no longer downloaded from profiles with pinned posts
- Reddit: bypass error 429 for saved posts
- Twitter: **data is not downloading due to domain change from twitter.com to x.com**
- Minor bugs
# 2024.5.4.0
*2024-05-04*
- Added
- YouTube (standalone app): setting to remove specific characters (`Defaults` - `Remove characters`)
- Instagram: simplify the `Connection closed` error
- Users search: add `Friendly name` to search results
- Fixed
- YouTube (standalone app): incorrect download processing when the file name ends with a dot (Issue #188)
- The program is freezes when editing users in some cases
- Sites
- Reddit: token update error
- Threads: unable to obtain credentials (`ID`)
# 2024.4.26.0
*2024-04-26*
- Added
- Site settings: the values that can be extracted from cookies immediately populate fields
- Feed: ability to load the last session of the current day (if it exists) as the current session after restarting SCrawler
- Users search: include friendly name matches in search result
- Updated
- gallery-dl up to version **1.26.9**
- Fixed
- xHamster: **saved posts aren't downloading**
# 2024.4.14.0
*2024-04-14*
- Fixed
- Facebook: can't get tokens
# 2024.4.13.0
*2024-04-13*
- Added
- Minor improvements
- PluginProvider
- IPluginContentProvider: added `ResetHistoryData` function
- Fixed
- Sites
- TikTok: remove last download date when erasing history data
- YouTube: remove last download date when erasing history data
- Instagram: **saved posts aren't downloading**
# 2024.4.10.0
*2024-04-10*
**For those of you who use TikTok, it is highly recommended to update TikTok plugin (using [these instructions](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-install-yt-dlp-ttuser-plugin)) to the latest version!**
**ATTENTION! This version includes changes to downloading groups (including the scheduler) and the settings file. Once you start using it, you won't be able to downgrade. I recommend making a backup of your SCrawler settings folder. It is also recommended to check all of your download groups settings and scheduler plans settings.**
- Added
- Sites
- TikTok: more settings for standalone downloader
- Instagram:
- automatically reset download options after updating credentials
- **ability to extract an image (if it exists) from a video that contains a static image**
- updated reels downloading function
- GraphQL support
- request timers (per request)
- OnlyFans:
- **download stories**
- option to disable timeline downloading
- Reddit: added `Check image` setting (unchecked by default) to make Reddit downloading faster
- **YouTube (standalone app)**:
- **the ability to add downloaded item(s) to a playlist**
- **the ability to create a playlist from downloaded music**
- calculation and display of the actual size of downloaded files
- **the ability to change audio bitrate** *(you can also set the default bitrate in `Defaults Audio` - `Bitrate`)*
- **embed thumbnail in the extracted audio (`mp3` only) as cover art** (Settings: `Defaults Audio` - `Embed thumbnail (extracted files)`)
- Exclude `drc` *(dynamic range compression)* from parsing results
- Standalone downloader:
- allow thumbnail to be saved with file
- calculation and display of the actual size of downloaded files
- Feed:
- add hotkeys: `Home`, `End`, `Up`, `Page Up`, `Down`, `Page Down`
- ability to save/load views
- Scheduler: **`All` and `Default` options removed.** *Only `Disabled`, `Specified` and `Groups` are available.*
- Download groups: **filter users who have been (not)downloaded in the last `x` days** *(download groups, advanced filter (main window), scheduler)*.
- Main window:
- ability to save/load views (`View` - `Save/Load view`)
- **all filter options from the `View` menu have now been moved to `Filter`**
- Settings:
- ability to confirm mass download using the `F6` key *(to protect against accidental pressing) (`Settings` - `Behavior`)*
- the ability to find the program environment
- default headers (`Settings` - `Headers`)
- Added the ability to display group users *(works in scheduler, scheduler plans, download groups)*
- Added exclusion of last 3 days from deleting session files
- Other improvements
- Updated
- **yt-dlp up to version 2024.04.09**
- PluginProvider
- Add `PropertyValueEventArgs` class
- PropertyValue:
- add `Checked` parameter
- add `OnCheckboxCheckedChange` handler
- ISiteSettings:
- add `CMDEncoding`, `EnvironmentPrograms` properties
- add `EnvironmentProgramsUpdated` function
- Fixed
- Sites
- PornHub: some videos won't download
- xHamster:
- some videos are missing when downloading creators
- user videos aren't downloading
- **JustForFans: fixed video downloading**
- TikTok (standalone downloader): files with long names aren't downloaded
- Users: incorrect decision to set last update date, which resulted in an incorrect last download date for some users
- Feed: a scrolling bug where the feed scrolls up after returning to it
- Minor bugs
# 2024.2.25.0
*2024-02-25*
- Added
- A `Feed` button has been added to notifications
- Feed:
- ability to merge multiple special feeds into one
- ability to select all/none media
- ability to add to a special feed(s) with removal from the current one
- the name of the loaded feed is now displayed in the form title
- `Refresh` button now refreshes the loaded feed
- ability to move/copy media
- Scheduler: the ability to move tasks (higher, lower) *(just a view attribute, doesn't affect the scheduler)*
- YouTube (standalone app): add `Open file` to the context menu
- YouTube (standalone app): **the ability to edit each playlist item**
- YouTube (standalone app): **embed thumbnail in the audio/video as cover art** (Settings: `Defaults Audio` - `Embed thumbnail`; `Defaults Video` - `Embed thumbnail (video)`)
- Instagram: the `csrftoken` can now be automatically extracted from cookies
- Instagram: remove `x-ig-www-claim` from settings
- Threads: the `csrftoken` can now be automatically extracted from cookies
- Threads: simplify 500 error when updating tokens
- Facebook: simplify token update errors
- OnlyFans: handle 500 error
- Plugins: added `ReplaceInternalPluginAttribute` attribute
- Other improvements
- Fixed
- Main window: incorrect sorting of profiles and collections
- Standalone downloader: url array form doesn't show scrollbars
- Feed: image rendering bug
- YouTube (standalone app): audio codec does not change when changing audio/video in the video options form
- Instagram: error downloading single post
- TikTok: files with long names aren't downloaded
- Minor bugs
# 2024.1.26.0
*2024-01-26*
- Added
- YouTube (standalone app): **the ability to reduce video FPS**
- TikTok: the ability to use a regex to clean the title
- YouTube (SCrawler): the ability to ignore community errors
- Fixed
- Instagram: stories (user) downloading with the wrong aspect ratio for some users
- Minor bugs
# 2024.1.20.0
*2024-01-20*
- Added
- Instagram: **the ability to download reels**
- LPSG: handle 404 error
# 2024.1.18.0
*2024-01-18*
- Fixed
- Main window: incorrect collection sorting
- xHamster: some user videos were not downloaded
- YouTube (standalone app): URL array form doesn't show scrollbars
- Minor bugs
# 2024.1.12.1
*2024-01-12*
- Added
- YouTube (SCrawler): data downloading by dates
- Feed: ability to merge multiple session feeds into one
- Feed: remove session number from special feeds
- Fixed
- **Instagram**: stories (user) downloading with the wrong aspect ratio for some users
- YouTube: incorrect opening of a post from the feed
- YouTube: wrong date to data parsing
# 2024.1.12.0
*2024-01-12*
- Added
- Feed: added a prompt before clearing the current session
- xHamster: creators
- YouTube communities: add error to log
- Added scheduler to tray menu
- Other improvements
- Fixed
- Feed: there is no option to create a new feed when adding checked items
- **Instagram**: downloading of tagged posts
- xHamster: profiles are not downloading
- Minor bugs
# 2023.12.27.0
*2023-12-27*
- Added
- Notification of new log data
- OnlyFans: **OF-Scrapper support to download DRM protected videos**
- Other improvements
- Fixed
- The default options are changed (`Favorite`, `Temporary`, etc.) when changing an account for a created user
- When changing the account for a created user, the new account does not apply to that user until SCrawler is restarted
- Saved posts: session file is not updated when new data is added
- Minor bugs
# 2023.12.15.0
*2023-12-15*
- Fixed
- Twitter: some twitter profiles don't download completely
- Minor bugs
# 2023.12.14.0
*2023-12-14*
- Added
- YouTube: options `Create thumbnail files (video)` and `Create thumbnail files (music)`
- YouTube: `Select all` and `Select none` buttons
# 2023.12.13.0
*2023-12-13*
- Added
- YouTube (standalone app): additional options for downloading channels
- Updated
- gallery-dl up to version 1.26.4
- Fixed
- Feed: saved posts are added to the end of the feed
- xHamster: some videos won't download
# 2023.12.10.0
*2023-12-10*
- Updated
- gallery-dl up to version 1.26.4-dev
- Fixed
- Twitter: data is not downloading
# 2023.12.7.0
*2023-12-07*
- Added
- Saved posts: add downloaded saved posts to the feed
- **YouTube (SCrawler): the ability to download YouTube user community feeds**
- Main window: add `Alt+A` hotkey to show scheduler
- Main window: add `Alt+P` hotkey to show progress form
- YouTube: check of adding a URL if it has already been downloaded
- YouTube: ability to check for a new version at start
- **Updater**
- Fixed
- Standalone downloader: URL files are not deleted along with the file
- Minor bugs
# 2023.11.25.0
*2023-11-25*
- Fixed
- Reddit: missing refresh token button in the settings form
# 2023.11.24.0
*2023-11-24*
For those of you who use TikTok, I recommend updating [TikTok plugin](https://github.com/bashonly/yt-dlp-TTUser) to the latest version using [these instructions](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-install-yt-dlp-ttuser-plugin).
- Added
- Automation: manual task option
- Scheduler: add scheduler name to form title
- Feeds: update when users' location and/or basic information changes
- Reddit: special notification for error 429
- TikTok: ID, username and friendly name extraction from data
- TikTok: new option `Use video date as file date`
- YouTube: absolute path for a single playlist
- Updated
- yt-dlp up to version 2023.11.16
- Fixed
- Scheduler: scheduler change error
- Twitter: JSON deserialization error
- xHamster, XVideos, PornHub, ThisVid: incorrect parsing of search queries
- YouTube: the file name is not changed manually
- YouTube: path not set when adding array to download
- Minor bugs
# 2023.11.17.0
*2023-11-17*
- Added
- **Facebook**
- **Multi-account**
- **Special feeds**
- Site settings: option `Download saved posts`
- Standalone downloader: support for multiple account
- PornHub: add playlists downloading
- YouTube: ability to download subtitles **and** `CC` if they both exists
- Other improvements
- PluginProvider
- `IDownloadableMedia`: added `AccountName` property
- `IPluginContentProvider`: added `AccountName` property
- `ISiteSettings`: added properties: `AccountName`, `Temporary`, `AvailableText`, `DefaultInstance`; added functions: `Clone`, `Update`, `Delete`; removed `Load` function; implement `IDisposable` interface
- `PropertyValue`: added functions: `BeginInit`, `EndInit`, `Clone`
- `Attributes.DoNotUse` - add `Value` field
- Fixed
- Instagram: handling 401 error
- OnlyFans: handling 401 error
- xHamster: handling 503 error
- xHamster: incorrect parsing of search queries
- XVideos: incorrect parsing of search queries
- ThisVid: incorrect parsing of search queries
- PornHub: incorrect parsing of search queries
- Automation: handle automation start error (in some cases) when changing scheduler
- Minor bugs
# 2023.10.10.0
*2023-10-10*
- Added
- Notification if the user has disabled downloading from the site
- Standalone downloader: new setting `Create URL files`
- Changed the sessions naming method to be more intuitive
- Settings that allow the user to change the number of saved session (`Settings` - `Feed` - `Store session data`)
- **YouTube: new settings `Create URL files` and `Create description files`**
- YouTube: added the `Clear selected` button
- YouTube: group the `Clear and remove` buttons in the menu
- Fixed
- Reddit: unable to save settings without OAuth data
- JustForFans: rewritten m3u8 downloader
- JustForFans: downloading of missing posts
- JustForFans: download to the date
- JustForFans: corrupted files
- Threads: new token is not saved if it was received during download
- ThisVid: parsing stops when new videos are added
- YouTube: file name is missing when destination is changed by selecting one of the saved locations
- YouTube: missing files still appear in the list
- Collections: labels are removed when creating a new collection
- Standalone downloader: cached thumbnail is not removed when item is removed from the list
- Minor bugs
# 2023.10.1.0
*2023-10-01*
- Added
- **Threads.net**
- YouTube: add URL standardization
- Fixed
- UserEditor: disable updating labels if they haven't changed
- Collections: incorrect updating of colors and labels when adding a new user
- RedGifs: incorrect handling of error 410
- Mastodon: hide error 503
- JustForFans: some profiles won't download
- Minor bugs
# 2023.9.21.0
*2023-09-21*
- Fixed
- PornHub: videos are not downloading
# 2023.9.20.0
*2023-09-20*
- Added
- **Instagram: user active (non-pinned) stories (Issue #17)**
- Reddit: reduce the number of token updates (refresh the token if there are Reddit users in the download queue)
- YouTube (standalone app): priority download protocol *(`Settings` - `Defaults` - `Protocol`)* (you can now select the default protocol you want to download media on: `Any`, `https`, `m3u8`))
- Automation: ability to change schedulers (`Download` - `Automation` - `Script icon`)
- Collections: update colors for the added user
- Fixed
- YouTube: can't detect `shorts` links
- Incorrect MD5 validation initial value
- Instagram: handle error 500
- Collections: update labels only for the added user
# 2023.8.27.0
*2023-08-27*
- Added
- **JustForFans**
- Advanced download (`Download` - `Download (advanced)`)
- Advanced filter (`View` - `Advanced filter`)
- Auto downloader: cloning plans
- Feed: add button to go to custom page
- Special log for non-existent users
- Twitter: group 'limit' notifications
- Ability to set custom color for subscription users
- Other improvements
- Fixed
- Auto downloader: new plan date display bug
- Auto downloader: downloading stuck
- Minor bugs
# 2023.8.6.0
*2023-08-06*
- Added
- The ability to remove user data and/or download history for redownload
- **Subscription** mode
- Settings to change the program title and information in the program information
- Settings for saving video thumbnail along with the file or in the cache (temporary cache or permanent cache)
- A bug report form to create a bug report or say something nice to the developer :blush:
- Prevent adding site-specific labels when adding to a collection
- Ability to select custom user highlighting in the main window and feed.
- Add a notification to the log if the user is not found on the site
- Added visualization of users download queue
- Ability to set more than one global paths
- Improve user paths changing: now you can also simply move the user/collection to another global location
- Ability to move multiple user/collection to another location
- Download groups: added `Subscription` options
- Download groups: the ability to set the number of users to download
- Auto downloader: new group options
- Auto downloader: additional skip options
- Auto downloader: added force start
- Feed: press `Ctrl+G` to go to a specific page
- Feed: added site icon to post
- Feed: always using `Friendly name` instead of `UserName` if it exists
- Missing posts: the ability to delete all missing posts
- Standalone downloader: add the ability to store download locations and quickly select after
- Standalone downloader: add `Ctrl+O` hotkey to select destination path
- Standalone downloader: add `Alt+O` hotkey to select destination path and save it to download locations
- User editor: ability to hide/show site-specific labels in collection editing mode
- Main window: filters by subscription and user
- Instagram: if the user is not found on the site, SCrawler will check for a new user name
- OnlyFans: handling of `504` and `429` errors
- OnlyFans: the `sec-ch-ua` header is now optional
- OnlyFans: ability to download 'Highlights" and media from chats
- PathPlugin: incorrect detection of path existence
- PornHub: completely rewritten videos parser
- PornHub: now you choose which videos you want to download (uploaded, tagged, private, favorites)
- PornHub: subscription mode
- PornHub: ability to download search queries and search categories
- Reddit: ability to set the number of concurrent downloads
- Reddit: added bearer token (optional)
- Reddit: added OAuth authorization (optional)
- Reddit: options to use the bearer token for the timeline and/or saved posts
- Reddit: option to disable the use of cookies for the timeline
- ThisVid: now you can also download user's favorite videos
- ThisVid: ability to download search queries, search categories and search tags
- ThisVid: subscription mode
- Twitter: new options: `Use the appropriate model`, `New endpoint: search`, `New endpoint: profiles`, `Abort on limit`, `Download already parsed` and `Media Model: allow non-user tweets`
- Twitter: new user option `Force apply`
- xHamster: ability to download search queries, search categories and search tags
- xHamster: subscription mode
- xHamster: pornstars download
- XVideos: ability to download search queries, search categories and search tags
- XVideos: subscription mode
- YouTube: added `Output path: ask for a name` and `Output path: auto add` settings
- YouTube: added the ability to store download locations and quickly select after
- YouTube: subscription mode
- Plugins.Attributes: added `DependentFields` attribute
- Plugins.Attributes: replace `Dependencies` with `Arguments` (`PropertyUpdater` attribute)
- Plugins.IPluginContentProvider: added `Options` and `IsSubscription` properties
- Plugins.ISiteSettings: added `SubscriptionsAllowed` property
- Plugins.ExchangeOptions: added `Options` field
- Plugins: added `ExitException`
- Other improvements
- Updated
- gallery-dl up to version 1.25.8
- yt-dlp up to version 2023.07.06
- LibVLCSharp up to 3.7.0
- VideoLAN up to 3.0.18
- Fixed
- **TikTok** supported again!
- Auto downloader: excluded labels and sites in default mode are not respected
- Download info: does not remember the last size and location
- Download info: hide unnecessary error
- Feed: `webm` photos not showing
- Search users: incorrect search by name
- OnlyFans: incorrect parsing of username containing dots
- OnlyFans: incorrect error handler
- Reddit: Handling error 502 (Reddit data not downloading)
- RedGifs: incorrect behavior when updating token
- Twitter: gifs are not downloading
- xHamster: some channels cannot be downloaded or are not fully downloaded
- YouTube: re-saving elements when loading a video list
- YouTube: files were not deleted when the delete button was clicked
- YouTube: a bug that caused the video to redownload
- Minor bugs
# 2023.6.19.0
*2023-06-19*
- Added
- **OnlyFans**
- YouTube: make the playlists parsing progress more informative
- YouTube: add `Add` button to tray
- YouTube: add `Ctrl+Click` on tray icon to add download
- YouTube: add setting `Download on click in tray: show form`
- Minor improvements to progress bars
- Other improvements
- Fixed
- YouTube: incorrect sorting algorithm
- LPSG: some files didn't download
- Reddit: downloaded gifs are static (Issue #141)
- xHamster: videos are not downloading or downloading incorrectly (Issue #144)
- Progress bar bugs
- Minor bugs
# 2023.6.9.0
*2023-06-09*
- Fixed
- YouTube: opening paths to downloaded playlists and channels
- Twitter: make the algorithm faster
- Make progress more informative
# 2023.6.8.0
*2023-06-08*
- Added
- YouTube: append artist name to music playlist output path
- YouTube: save thumbnail path for playlist and channel
- Fixed
- YouTube: opening paths to downloaded playlists and channels
- Twitter: profile not fully downloaded
- Corrected form size for small monitors (Issue #136)
# 2023.6.5.0
*2023-06-05*
- Added
- **Instagram**: add additional authorization headers
- Setting to prevent user icon and banner from downloading (Request #129)
- Add standalone downloader to tray context menu
- YouTube downloader: added `Replace modification date` property
- Minor improvements
- Fixed
- Fascist **Twitter**: posts not downloading (new API)
- Main window: refill bug when the number of filtered profiles = 0
- Standalone downloader: new items are not added to the queue
- Standalone downloader: bug when not downloaded videos do not appear in the list when loading the program
- Standalone downloader: add videos array not working
- Saved posts: remove main progress perform when downloading saved posts
- Minor bugs
# 2023.5.12.0
*2023-05-12*
- Added
- Advanced progress (make progress bars more informative)
- User metrics calculation
- Reddit: improve parsing function
- PornHub: add `Download UHD` option
- Fixed
- MD5 GIF hash bug
- Mastodon: handle 'Forbidden' error
- Mastodon: bug in parsing non-user posts
- Pinterest: remove cookies requirement for saved posts
- PornHub: resolutions issue
- Reddit: missing & broken images bug
- Main window: collection pointing bug
# 2023.4.28.0
*2023-04-28*

162
FAQ.md
View File

@@ -1,116 +1,100 @@
# Frequently asked questions
**Please read the [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) Before asking a question!**
**Join our Discord server**: https://discord.gg/uFNUXvFFmg
<br/>*You can get help faster there!*
**Also read [here](README.md) for basic information.**
# Docs
- Basic info: https://github.com/AAndyProgram/SCrawler/blob/main/README.md
- **GUIDE**: https://github.com/AAndyProgram/SCrawler/wiki/
- Settings: https://github.com/AAndyProgram/SCrawler/wiki/Settings
- Discord: https://discord.gg/uFNUXvFFmg
Most of your questions are already answered. All settings, functions, buttons and everything else described in the guide.
Any other questions I will keep in this file.
# Backup
I strongly recommend you to **regularly** create backup copies of the settings files. **An [example script](https://github.com/AAndyProgram/SCrawler/blob/main/Tools/ArchiveSCrawlerUsersDataFiles.bat) for this** on GitHub (you **should adapt** it to your environment, and you can use it when [SCrawler is closed](https://github.com/AAndyProgram/SCrawler/wiki/Settings#behavior)).
----
**This way you'll always have the latest backup of your settings files and can restore it if something goes wrong!**
#### Q: **HOW TO SETUP COOKIES**
A: https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies
<!---**ATTENTION! If you need to use cookies but cannot import them, I highly recommend that you don't use SCrawler and use another program. Don't create issues, discussions, or write to me on Discord. Any issue or discussion about cookies will be deleted immediately without a response. Any user who asks me about cookies on Discord will be banned.**--->
----
#### Q: **Does this program have GUI or CLI.**
A: This is a GUI program.
----
#### Q: **Will CLI be added in the future?**
A: NO.
----
#### Q: **I want to add "...." site. How to request.**
<!---A: How to request a new site you can read [here](CONTRIBUTING.md#how-to-request-a-new-site)--->
**I'm currently not accepting requests to develop new sites.**
----
#### Q: **Site download failed.**
A: Check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set, but nothing works, go to [create a new issue](https://github.com/AAndyProgram/SCrawler/issues). Don't forget to attach the LOG.
# How to report a problem
1. **Post your problem [here](https://github.com/AAndyProgram/SCrawler/issues) or in the [help channel](https://discord.com/channels/1124032649682493462/1124281838056259614) on our Discord server**
2. Attach the **profile URLs or links** that you cannot download.
3. Attach the **LOG** if it exists.
4. Attach **the environment information** copied from SCrawler (click the top right info button in the main window, then the `Environment` button, then the `Copy` button, and paste the copied text into the message).
5. *Add screenshots to illustrate the problem (**optional**)*
**ATTENTION! Issues without URLs will be closed without a response!**
----
# Most frequently questions about SCrawler
#### Q: **I have set credentials but still nothing is downloading**
**If something doesn't download, always check the [SITE'S REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements) before asking questions!**
A: Click the ```Start downloading``` button
*How to use: find your problem in the list and read the answer.*
----
## General questions
- **PROFILES**
- I added a profile but **nothing downloaded** :arrow_forward: check your cookies and [site requirements](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements). If there are any optional fields that you don't fill in, do so. Still nothing works - [report it](#how-to-report-a-problem)!
- User downloading failed :arrow_forward: check your credentials and **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**. If all settings are set and nothing works, [report it](#how-to-report-a-problem). **Don't forget to attach the LOG.**
- [How to redownload user](https://github.com/AAndyProgram/SCrawler/wiki#redownload-user)
- How to **add profile** to download :arrow_forward: copy the **[profile URL](https://github.com/AAndyProgram/SCrawler/wiki#add-user)** and press `Insert` or `Ctrl+Insert`. **ALWAYS PASTE THE USER PROFILE URL**. After that select this user and press `F5` or click the `Download selected` button.
- How to download **[saved posts](https://github.com/AAndyProgram/SCrawler/wiki#saved-posts)**
- **[HOW TO ADD COOKIES](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-set-up-cookies)**
- [How to report a problem](#how-to-report-a-problem)
- I want you to **add the site** to SCrawler :arrow_forward: **I'm not currently accepting requests to add new sites**, but you can [create a plugin](https://github.com/AAndyProgram/SCrawler/wiki/Plugins) (for your site) for SCrawler.
- What language is SCrawler written in :arrow_forward: `vb.net`
- I don't know `vb.net` and I can't write a plugin :arrow_forward: you can write a plugin in `C#`
- I have a suggestion, will it be added :arrow_forward: maybe if it interested me.
- How to name files using a pattern (e.g. `Site_PostID_Name.jpg`) :arrow_forward: **there is no such functionality and there are no such plans**.
- **DON'T CHANGE THE DEFAULT SITE SETTINGS UNLESS YOU KNOW EXACTLY WHAT YOU'RE DOING!** SCrawler already has all the default settings to work. You only need to add credentials (where [required](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)).
- My computer shut down while SCrawler was running and now **SCrawler won't start or some users are missing** :arrow_forward: restore user settings from [backup](#backup).
- Installation, update and configuration
- How to install: https://github.com/AAndyProgram/SCrawler#installation
- How to update: https://github.com/AAndyProgram/SCrawler#updating
- What file executes the program: **`SCrawler.exe`**
- Where to find binaries: https://github.com/AAndyProgram/SCrawler/releases/latest
- [How to build from source](https://github.com/AAndyProgram/SCrawler/blob/main/CONTRIBUTING.md#how-to-build-from-source)
- [Video how to configure](#video-how-to-configure)
- **Antivirus**
- **Antivirus detects SCrawler as a virus** :arrow_forward: SCrawler doesn't contain any viruses at all. All code is posted on GitHub. You can review it. I have nothing to hide. SCrawler just downloads pictures and videos. That's all. If you trust SCrawler, you should just add it to the antivirus exceptions, as I did. Sometimes antiviruses identify SCawler as a virus. This is usually related to the number of files being edited (users' settings files) and the number of files being downloaded. In this case, the antivirus can also remove these files, which will damage users' settings. **If you don't trust SCrawler, just delete it.**
- **Antivirus detects gallery-dl as a virus** :arrow_forward: it's a trustworthy program that is trusted by thousands of people around the world. Antiviruses identify some builds as containing viruses, but this is not true. **If you don't trust gallery-dl, you can simply delete it. But if you delete it, you won't be able to download [Twitter & Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#gallery-dl).** You should decide for yourself.
#### Q: **Where can I find the release?**
## Sites questions
A: https://github.com/AAndyProgram/SCrawler/releases/latest
*How to use: find the site you need in the list and read the answer.*
----
- Reddit: don't use credentials at all or configure [OAuth](https://github.com/AAndyProgram/SCrawler/wiki/Settings#how-to-get-reddit-credentials). **Reddit profiles can be downloaded without any credentials at all. Subreddits require OAuth! If nothing downloads, use OAuth!** Don't use OAuth token to download saved posts (use cookies only).
- **META** (**Instagram**, Threads, Facebook): you need **cookies** and fill in **all fields**
- **Instagram [TIPS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram-tips)**
- **Instagram saved posts**: I don't consider questions like "I have 10k saved posts and only 1000 were downloaded". Download posts, remove them from saved posts, delete the `Saved posts` **settings folder**, repeat.
- TikTok: works via yt-dlp. If something doesn't download, we need to wait until yt-dlp fixes it. TikTok doesn't require cookies to download.
- Porn sites: **COOKIES**!
- ThisVid: https://github.com/AAndyProgram/SCrawler/wiki/Settings#thisvid-faq
- **OnlyFans**: cookies + **all fields** + [OF-Scraper (download the correct version that I pointed)](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper) & [mp4decrypt](https://www.bento4.com/downloads/) & **DRM keys** to download DRM protected videos. [OF-Scraper support](https://github.com/AAndyProgram/SCrawler/wiki/Settings#of-scraper-support). Also read [this](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans-faq)
- **JustForFans**: **THE VIDEO ISN'T DOWNLOADING AT THE MOMENT** ([Issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870))
#### Q: **How to run the program?**
## Other questions
A: Double-click ```SCrawler.exe```
### Does the program remember the last download and check for new posts, downloading only new posts, or does the program download the entire profile every time
The program stored posts IDs in users' folders. For the first time, the program downloads the entire profile. All subsequent times the program will check for new posts and download **only new posts**!
----
### Does this program have a GUI or CLI, and will a CLI be added in the future
This is a GUI program and **NO**, <u>CLI will not be added</u>
#### Q: **Where to find binaries?**
### How to remove the label
There is no functionality to remove an individual label. You can open the `Labels.txt` file in the program settings folder and delete any label you want. You also can delete this file (`Labels.txt`). In this case, when SCrawler is launched, the list of labels will be populated only with existing labels (from the user data files).
A: https://github.com/AAndyProgram/SCrawler/releases/latest
### How to remove a user from the blacklist
Just add that user back to the program. In the dialog box that opens, click the `Add and remove from blacklist` button.
----
### You lost me. Your program is too complicated.
**I'm fine with that**. If the program is too complicated for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I develop SCrawler for myself and publish it on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm okay with it.
#### Q: **Does the program remember the last download and check for new posts, downloading only new posts? Or does the program download the entire profile every time?**
### Add a step-by-step guide or video on how to use the program
**NO!** The guide fully covers all the functionality of SCrawler! If you don't respect my work, I don't waste my time. If you want, you can create a video tutorial and send it to me. Then I'll add it. All options and their purposes are described on the wiki. The wiki also contains a description of all the settings and how to configure them. For complex settings there is a step-by-step guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I've developed a program with an intuitive interface. There is a `Settings` button, download buttons, a context menu that appears when you right-click on a user, and other controls. Anyone can use it.
A: The program stored posts IDs in users' folders. For the first time, the program downloads the entire profile. All subsequent times the program will check for new posts and download **only new posts**!
**There is already a [video](#video-how-to-configure) example of how to configure a site.**
----
# Video how to configure
#### Q: **How to redownload all data**
**The following video was recorded by a user who loves SCrawler and demonstrates how to add credentials using Instagram as an example:**
A: Double-click on the user you want to redownload. In the opened window open folder setting. Delete the files ending with ```_Data.xml``` and ```_Posts.txt```. Restart SCrawler. Download this user again.
----
#### Q: **How to remove the label**
A: There is no functionality to remove an individual label. You can open the ```Labels.txt``` file in the program settings folder and delete any label you want. You also can delete this file (```Labels.txt```). In this case, when the program starts, the list of labels list will be updated with only existing labels (from the user data files).
----
#### Q: **How to remove a user from the blacklist**
A: Just add that user back to the program. In the dialog box that opens, click on the ```Add and remove from blacklist``` button.
----
#### Q: **Why don't you answer how it works**
A: Because **I don't want to**. I don't want to waste my time explaining things that are already covered in the **[GUIDE](https://github.com/AAndyProgram/SCrawler/wiki)**! If you didn't bother to read the guide, why would I waste my time?! ALL FUNCTIONALITY IS DESCRIBED IN THE GUIDE. Before publishing a new release, I update the guide. If you don't respect my work, I don't waste my time.
----
#### Q: **You lost me. Your program is too complicated.**
A: **I'm fine with that**. If the program is difficult for you or you can't configure it, I can only suggest you find another (easier) program. I really don't mind! The program is free. I am develop SCrawler for myself and publish on GitHub because people found my program useful. If someone can't use it or doesn't like it, I'm fine.
----
#### Q: **I can't configure something**
A: I can only [suggest](#q-you-lost-me-your-program-is-too-complicated) you find another (easier) program.
----
#### Q: **Can you add a step-by-step guide or video on how to use the program?**
A: **NO! NEVER!** The guide fully covers all the functionality of SCrawler! If you don't respect my work, I don't waste my time. If you want, you can create a video tutorial and send it to me. Then I add it. All options and what each option does described on the wiki. The wiki also contains a description of all settings and how-to configure them. For complex settings, there is a steep-by-steep guide. Read the [main](README.md) information and [GUIDE](https://github.com/AAndyProgram/SCrawler/wiki/) and you won't have any problems. I have developed a program with an intuitive interface. There is a Settings button, download buttons, a context menu that drops down when a user is clicked, and other controls. Anyone can use it.
[![How to configure](https://img.youtube.com/vi/XDn7zG4I700/0.jpg)](https://www.youtube.com/watch?v=XDn7zG4I700)

View File

@@ -1,3 +1,9 @@
You can create a plugin for any site you want. **To create a plugin, read [this guide](https://github.com/AAndyProgram/SCrawler/wiki/Plugins).**
If you've created a plugin, you can create a [new issue](https://github.com/AAndyProgram/SCrawler/issues/new?assignees=&labels=New+Plugin&projects=&template=plugin_add.md&title=%5BNEW+PLUGIN%5D) and I'll add your plugin to the list below.
----
List of available plugins:
Tools:

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 56 KiB

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 483 KiB

After

Width:  |  Height:  |  Size: 472 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.7 KiB

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.7 KiB

After

Width:  |  Height:  |  Size: 101 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.2 KiB

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 363 KiB

After

Width:  |  Height:  |  Size: 359 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.5 KiB

After

Width:  |  Height:  |  Size: 6.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.9 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 12 KiB

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 12 KiB

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

After

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 12 KiB

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.3 KiB

After

Width:  |  Height:  |  Size: 7.7 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 12 KiB

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 27 KiB

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 19 KiB

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 19 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.0 KiB

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.3 KiB

After

Width:  |  Height:  |  Size: 5.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.8 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

View File

@@ -6,11 +6,19 @@ https://github.com/yt-dlp/yt-dlp/
SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.
# gallery-dl
https://github.com/mikf/gallery-dl
**Great powerful CLI tool that supports hundreds of sites.**
SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.
# 4K Video Downloader
https://www.4kdownload.com/-plbrz/video-downloader
| Option | SCrawler | 4K Stogram |
| Option | SCrawler | 4K Video Downloader |
| ---- | ---- | ---- |
| User managament | **Advanced** | No |
| Automatic downloads | **Yes** | No |
@@ -117,14 +125,7 @@ https://github.com/RipMeApp/ripme
| **Free options** | The program is completely free | The program is completely free, but site limits are not declared |
| Operating Systems | Windows 10+ | Windows, MacOS, Linux |
| Select want content type to download | Yes | Yes |
| Suported sites | 15 internal and any site using plugins | 86+ sites (declared) |
| Suported sites | 15+ internal and any site using plugins | 86+ sites (declared) |
| Other sites support | **Yes** | No |
| Still supported | **Yes** | **No (last release date May 4, 2021)** |
# gallery-dl
https://github.com/mikf/gallery-dl
**CLI tool**
SCrawler has advanced user management, collections, labels, groups, automatic downloads, a beautiful view, GUI, the ability to add plugins for other sites and much more. Just try it and compare.

131
README.md
View File

@@ -1,7 +1,10 @@
# :rainbow_flag: Social networks crawler :rainbow_flag:
# 🏳️‍🌈 Happy LGBT Pride Month 🎉
# 🏳️‍🌈 Social networks crawler 🏳️‍🌈
[![GitHub release (latest by date)](https://img.shields.io/github/v/release/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/releases/latest)
[![GitHub license](https://img.shields.io/github/license/AAndyProgram/SCrawler)](https://github.com/AAndyProgram/SCrawler/blob/main/LICENSE)
[![Discord](https://img.shields.io/discord/1124032649682493462?logo=discord)](https://discord.gg/uFNUXvFFmg)
[![GitHub all releases](https://img.shields.io/github/downloads/aandyprogram/scrawler/total?label=Total%20downloads)](https://github.com/AAndyProgram/SCrawler/releases)
[![FAQ](https://img.shields.io/badge/FAQ-green)](FAQ.md)
[![GUIDE](https://img.shields.io/badge/GUIDE-green)](https://github.com/AAndyProgram/SCrawler/wiki)
@@ -9,46 +12,59 @@
:eu:
:greece:
A program to download photo and video from [any site](#supported-sites) (e.g. YouTube, YouTube Music, Reddit, Twitter, Mastodon, Instagram, TikTok, RedGifs, PornHub, XHamster, XVIDEOS, ThisVid, LPSG, Pinterest).
A program to download photo and video from [any site](#supported-sites) (e.g. YouTube, YouTube Music, OnlyFans, Reddit, Twitter, Mastodon, Instagram, Threads, Facebook, TikTok, RedGifs, JustForFans, PornHub, XHamster, XVIDEOS, ThisVid, LPSG, Pinterest).
**If you like SCrawler, please like the program on [this site](https://alternativeto.net/software/scrawler/about/) and/or [this](https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml)**
**Join our Discord server**: https://discord.gg/uFNUXvFFmg
<br/>*If you have problems using the program, you can get help faster on our Discord server!*
<!---Do you like this program? Consider adding to my coffee fund by making a donation to show your support. :blush:
[![ko-fi](https://www.ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/andyprogram)--->
**Bitcoin**: BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET
[![](https://www.softpedia.com/_img/softpedia_100_free.png?2023_1)](https://www.softpedia.com/get/Internet/Download-Managers/Social-networks-crawler.shtml#status)
![Main window](ProgramScreenshots/MainWindow.png)
![Channels window](ProgramScreenshots/Channels.png)
[**YouTube standalone application:**](https://github.com/AAndyProgram/SCrawler/wiki/YouTube%20downloader)
[**YouTube standalone application:**](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader)
![YouTube application](ProgramScreenshots/AppYouTube.png)
# What can program do:
- Download pictures and videos from users' profiles and subreddits:
- YouTube videos, shorts, users, artists, playlists, music, tracks;
- Reddit images, galleries of images, videos, saved posts;
- Redgifs videos (https://www.redgifs.com/);
- Twitter images and videos, saved (bookmarked) posts;
- Download pictures and videos from user profiles:
- YouTube videos, shorts, community feeds, users, artists, playlists, music, tracks;
- Reddit images, galleries of images, videos, text, saved posts;
- Redgifs images and videos (https://www.redgifs.com/);
- Twitter images and videos, text, saved (bookmarked) posts, likes, communities;
- Bluesky images and videos, text;
- OnlyFans images and videos, text, saved (bookmarked) posts, stories;
- JustForFans images and videos, saved (bookmarked) posts;
- Mastodon images and videos, saved (bookmarked) posts;
- Instagram images and videos, tagged posts, stories, saved posts;
- TikTok videos (*currently broken*; [limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits));
- Instagram images and videos, text, tagged posts, stories, saved posts;
- Threads images and videos, text, saved posts;
- Facebook images and videos, stories, saved posts;
- TikTok images and videos;
- Pinterest boards, users, saved posts;
- Imgur images, galleries and videos;
- Gfycat videos;
- PornHub images, videos, save (liked) posts;
- XHamster images, videos, saved posts;
- XVIDEOS videos, saved posts;
- ThiVid images, videos, saved posts;
- PornHub images, videos, save (liked) posts, search queries, search categories;
- XHamster images, videos, saved posts, search queries, search categories, search tags;
- XVIDEOS videos, saved posts, search queries, search categories;
- ThisVid images, videos, saved posts, search queries, search categories, search tags;
- [Other](#supported-sites) supported sites
- Parse [channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
- Download [saved Reddit, Twitter and Instagram posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
- Parse [Reddit channel and view data](https://github.com/AAndyProgram/SCrawler/wiki/Channels)
- Download [saved posts](https://github.com/AAndyProgram/SCrawler/wiki/Home#saved-posts)
- Add users from parsed channel
- **Advanced user management**
- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every ```X``` minutes)
- **Feed** ([feed](https://github.com/AAndyProgram/SCrawler/wiki#feed) of downloaded media files)
- **Automation** ([downloading data automatically](https://github.com/AAndyProgram/SCrawler/wiki/Settings#automation) every `X` minutes)
- **Feed** ([feed](https://github.com/AAndyProgram/SCrawler/wiki#feed) of downloaded media files and subscriptions posts)
- Multiple accounts support
- Labeling users
- Create [download groups](https://github.com/AAndyProgram/SCrawler/wiki/Settings#download-groups)
- Adding users to favorites and temporary
- Adding users and search queries in the **Subscription** mode (download post preview, but do not download the media file)
- [Filter exists users](https://github.com/AAndyProgram/SCrawler/wiki#view) by label or group
- Selection of media types you want to download (images only, videos only, both)
- [Download a special video](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video), image or gallery
@@ -63,12 +79,17 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo
- **YouTube Music**
- **Reddit**
- **Twitter**
- **Mastodon**
- **Bluesky**
- **OnlyFans** *(partial support)*[^1]
- **Instagram**
- TikTok (*currently broken*; [limited](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok-limits))
- **Threads**
- **Facebook**
- JustForFans *(partial support) ([video issue](https://discord.com/channels/1124032649682493462/1205547615199039551/1231349555132366870))*[^1]
- Mastodon *(out of support)*
- TikTok
- RedGifs
- Pinterest
- Imgur
- Imgur *(out of support)*
- Gfycat
- LPSG
- **PornHub**
@@ -83,14 +104,6 @@ A program to download photo and video from [any site](#supported-sites) (e.g. Yo
First, the program downloads the full profile. After the program downloads only new posts. The program remembers downloaded posts.
## Reddit
The program parses user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.
## Other sites
The program parses user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.
## How to request a new site
<!---Read [here](CONTRIBUTING.md#how-to-request-a-new-site) about--->
@@ -98,7 +111,7 @@ The program parses user posts and compares file names with existing ones to remo
# Requirements
- Windows 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
- **Windows 10, 11** with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this [tool](Tools/NET.FrameworkVersion.ps1).
- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
# Guide
@@ -120,29 +133,44 @@ The program parses user posts and compares file names with existing ones to remo
- **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
- [Reddit](https://github.com/AAndyProgram/SCrawler/wiki/Settings#reddit)
- [Twitter](https://github.com/AAndyProgram/SCrawler/wiki/Settings#twitter)
- [Mastodon](https://github.com/AAndyProgram/SCrawler/wiki/Settings#Mastodon)
- [Bluesky](https://github.com/AAndyProgram/SCrawler/wiki/Settings#bluesky)
- [OnlyFans](https://github.com/AAndyProgram/SCrawler/wiki/Settings#onlyfans)
- [Mastodon](https://github.com/AAndyProgram/SCrawler/wiki/Settings#mastodon)
- [Instagram](https://github.com/AAndyProgram/SCrawler/wiki/Settings#instagram)
- [Threads](https://github.com/AAndyProgram/SCrawler/wiki/Settings#threads)
- [Facebook](https://github.com/AAndyProgram/SCrawler/wiki/Settings#facebook)
- [JustForFans](https://github.com/AAndyProgram/SCrawler/wiki/Settings#justforfans)
- [TikTok](https://github.com/AAndyProgram/SCrawler/wiki/Settings#tiktok)
- [RedGifs](https://github.com/AAndyProgram/SCrawler/wiki/Settings#redgifs)
- [YouTube](https://github.com/AAndyProgram/SCrawler/wiki/Settings#YouTube)
- [YouTube](https://github.com/AAndyProgram/SCrawler/wiki/Settings#youtube)
- [Pinterest](https://github.com/AAndyProgram/SCrawler/wiki/Settings#Pinterest)
- [PornHub](https://github.com/AAndyProgram/SCrawler/wiki/Settings#pornhub)
- [XHamster](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xhamster)
- [XVIDEOS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#xvideos)
- [ThisVid](https://github.com/AAndyProgram/SCrawler/wiki/Settings#ThisVid)
- [ThisVid](https://github.com/AAndyProgram/SCrawler/wiki/Settings#thisvid)
- [LPSG](https://github.com/AAndyProgram/SCrawler/wiki/Settings#lpsg)
**Full guide you can find [here](https://github.com/AAndyProgram/SCrawler/wiki)**
**Video on how to configure**
[![How to configure](https://img.youtube.com/vi/XDn7zG4I700/0.jpg)](https://www.youtube.com/watch?v=XDn7zG4I700)
# Installation
**Just download the [latest release](https://github.com/AAndyProgram/SCrawler/releases/latest), unzip the program archive to any folder and enjoy.** :blush:
**Don't put program in the ```Program Files``` system folder (this is portable program and program settings are stored in the program folder)**
**Don't put program in the `Program Files` system folder (this is portable program and program settings are stored in the program folder)**
**I highly doubt you can run SCrawler on Linux or Mac. SCrawler is a program that is heavily dependent on Windows.**
# Updating
Just download [latest](https://github.com/AAndyProgram/SCrawler/releases/latest) version and unpack it into the program folder. **Before starting a new version, I recommend making a backup copy of the program settings folder.**
Just download [latest](https://github.com/AAndyProgram/SCrawler/releases/latest) version and unpack it into the program folder. **Before launching a new version, I recommend making a backup copy of the program settings folder and user settings/data files.**
**You can also use the updater included in the release package.**
# [How to report a problem](CONTRIBUTING.md#how-to-report-a-problem)
# [How to build from source](CONTRIBUTING.md#how-to-build-from-source)
@@ -156,13 +184,38 @@ The program has an intuitive interface.
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
Just add a user profile and **click the ```Download``` button**.
[![How to configure](https://img.youtube.com/vi/XDn7zG4I700/0.jpg)](https://www.youtube.com/watch?v=XDn7zG4I700)
Read more about adding users and subreddits [here](https://github.com/AAndyProgram/SCrawler/wiki#Add%20user)
Just add a user profile and **click the `Download` button**.
```mermaid
stateDiagram
Start: Add site credentials
What: What would I like to do
DownUser: Download user
DownVideo: Download video
AUser: Add user (1)
OVIF: Open standalone downloader (2)
AVideo: Add video url
F5: Press 'F5' or click the download button
[*]-->Start
Start-->What
What-->DownUser
What-->DownVideo
DownUser-->AUser
DownVideo-->OVIF
OVIF-->AVideo
AVideo-->F5
AUser-->F5
F5-->[*]
```
1. Press `Insert` or click the `Download` button ([read more here](https://github.com/AAndyProgram/SCrawler/wiki#users-list), [hot keys](https://github.com/AAndyProgram/SCrawler/wiki#hot-keys))
2. Click the `Download` button, then `Standalone downloader` ([read more here](https://github.com/AAndyProgram/SCrawler/wiki#download-separate-video))
![Add user](ProgramScreenshots/CreateUserClear.png)
# Contact me
Matrix (Element): https://matrix.to/#/@andyprogram:matrix.org
Discord: AndyProgram#3804
Discord server: https://discord.gg/uFNUXvFFmg
[^1]: Partial support means that I don't have personal accounts on paid porn sites because I don't pay for porn. If this site has stopped downloading and you want me to fix it, please be ready to give me access to an account with at least one active subscription. Otherwise, the download from this site will not be fixed.

View File

@@ -1,3 +1,3 @@
[*.vb]
# Modifier preferences
file_header_template = Copyright (C) 2023 Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>
file_header_template = Copyright (C) Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>

View File

@@ -36,17 +36,44 @@ Namespace Plugin.Attributes
Public Property IsInformationLabel As Boolean = False
''' <summary>Label text alignment.<br/>Default: <see cref="Drawing.ContentAlignment.TopCenter"/></summary>
Public Property LabelTextAlign As Drawing.ContentAlignment = Drawing.ContentAlignment.TopCenter
Private _IsAuth As Boolean = False
''' <summary>This is an authorization property</summary>
Public Property IsAuth As Boolean = False
Public Property IsAuth As Boolean
Get
Return _IsAuth
End Get
Set(ByVal _IsAuth As Boolean)
Me._IsAuth = _IsAuth
If _IsAuth And String.IsNullOrEmpty(Category) Then
Category = CategoryAuth
ElseIf Not _IsAuth AndAlso Not String.IsNullOrEmpty(Category) AndAlso Category = CategoryAuth Then
Category = String.Empty
End If
End Set
End Property
Public Const CategoryAuth As String = "Authorization"
Public Property Category As String = Nothing
Public Property InheritanceName As String = Nothing
''' <summary>Initialize a new property option attribute</summary>
''' <param name="PropertyName">Property name</param>
Public Sub New(<CallerMemberName()> Optional ByVal PropertyName As String = Nothing)
Name = PropertyName
End Sub
End Class
''' <summary>Set the dependent fields that need to be updated when this property is changed internally.</summary>
<AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class DependentFields : Inherits Attribute
Public ReadOnly Fields As String()
Public Sub New(ByVal Field As String)
Fields = {Field}
End Sub
Public Sub New(ByVal Fields As String())
Me.Fields = Fields
End Sub
End Class
''' <summary>Store property value in settings XML file</summary>
<AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class PXML : Inherits Attribute
Public ReadOnly ElementName As String
Public Property OnlyForChecked As Boolean = False
''' <summary>Initialize a new XML attribute</summary>
''' <param name="XMLElementName">XML element name</param>
Public Sub New(<CallerMemberName()> Optional ByVal XMLElementName As String = Nothing)
@@ -55,20 +82,26 @@ Namespace Plugin.Attributes
End Class
''' <summary>Attribute to disable some properties for host use</summary>
<AttributeUsage(AttributeTargets.Property, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class DoNotUse : Inherits Attribute
Public ReadOnly Value As Boolean = True
Public Sub New()
End Sub
Public Sub New(ByVal Value As Boolean)
Me.Value = Value
End Sub
End Class
''' <summary>Special property updater</summary>
<AttributeUsage(AttributeTargets.Method, AllowMultiple:=True, Inherited:=False)> Public NotInheritable Class PropertyUpdater : Inherits Attribute
Public ReadOnly Name As String
Public ReadOnly Dependencies As String()
Public ReadOnly Arguments As String()
''' <inheritdoc cref="PropertyUpdater.New(String, String())"/>
Public Sub New(ByVal UpdatingPropertyName As String)
Name = UpdatingPropertyName
End Sub
''' <summary>Initialize a new PropertyUpdater attribute</summary>
''' <param name="UpdatingPropertyName">The name of the property to be updated</param>
Public Sub New(ByVal UpdatingPropertyName As String, ByVal Dependent As String())
Public Sub New(ByVal UpdatingPropertyName As String, ByVal Arguments As String())
Name = UpdatingPropertyName
Dependencies = Dependent
Me.Arguments = Arguments
End Sub
End Class
''' <summary>Plugin key</summary>
@@ -172,4 +205,13 @@ Namespace Plugin.Attributes
Repository = RepoName
End Sub
End Class
''' <summary>Replace internal plugin with the current one</summary>
<AttributeUsage(AttributeTargets.Class, AllowMultiple:=False, Inherited:=False)> Public NotInheritable Class ReplaceInternalPluginAttribute : Inherits Attribute
Public ReadOnly SiteName As String
Public ReadOnly PluginKey As String
Public Sub New(ByVal PluginKey As String, Optional ByVal SiteName As String = Nothing)
Me.PluginKey = PluginKey
Me.SiteName = SiteName
End Sub
End Class
End Namespace

View File

@@ -14,6 +14,7 @@ Namespace Plugin
ReadOnly Property SiteIcon As Drawing.Image
ReadOnly Property Site As String
ReadOnly Property SiteKey As String
Property AccountName As String
Property ThumbnailUrl As String
Property ThumbnailFile As String
Property Title As String

View File

@@ -10,11 +10,16 @@ Namespace Plugin
Public Interface IPluginContentProvider : Inherits IDisposable
Event ProgressChanged(ByVal Value As Integer)
Event ProgressMaximumChanged(ByVal Value As Integer, ByVal Add As Boolean)
Event ProgressPreChanged As ProgressChangedEventHandler
Event ProgressPreMaximumChanged As ProgressMaximumChangedEventHandler
Property Thrower As IThrower
Property LogProvider As ILogProvider
Property Settings As ISiteSettings
Property AccountName As String
Property Name As String
Property NameTrue As String
Property ID As String
Property Options As String
Property ParseUserMediaOnly As Boolean
Property UserDescription As String
Property ExistingContentList As List(Of IUserMedia)
@@ -23,6 +28,7 @@ Namespace Plugin
Property UserExists As Boolean
Property UserSuspended As Boolean
Property IsSavedPosts As Boolean
Property IsSubscription As Boolean
Property SeparateVideoFolder As Boolean
Property DataPath As String
Property PostsNumberLimit As Integer?
@@ -35,5 +41,6 @@ Namespace Plugin
Sub GetMedia(ByVal Token As Threading.CancellationToken)
Sub Download(ByVal Token As Threading.CancellationToken)
Sub DownloadSingleObject(ByVal Data As IDownloadableMedia, ByVal Token As Threading.CancellationToken)
Sub ResetHistoryData()
End Interface
End Namespace

View File

@@ -8,7 +8,7 @@
' but WITHOUT ANY WARRANTY
Imports System.Drawing
Namespace Plugin
Public Interface ISiteSettings
Public Interface ISiteSettings : Inherits IDisposable
Enum Download As Integer
Main = 0
SavedPosts = 1
@@ -17,6 +17,14 @@ Namespace Plugin
ReadOnly Property Icon As Icon
ReadOnly Property Image As Image
ReadOnly Property Site As String
Property CMDEncoding As String
Property EnvironmentPrograms As IEnumerable(Of String)
Property UserAgentDefault As String
Sub EnvironmentProgramsUpdated()
Property AccountName As String
Property Temporary As Boolean
Property DefaultInstance As ISiteSettings
ReadOnly Property SubscriptionsAllowed As Boolean
Property Logger As ILogProvider
Function GetUserUrl(ByVal User As IPluginContentProvider) As String
Function IsMyUser(ByVal UserURL As String) As ExchangeOptions
@@ -24,9 +32,6 @@ Namespace Plugin
Function GetInstance(ByVal What As Download) As IPluginContentProvider
Function GetSingleMediaInstance(ByVal URL As String, ByVal OutputFile As String) As IDownloadableMedia
Function GetUserPostUrl(ByVal User As IPluginContentProvider, ByVal Media As IUserMedia) As String
#Region "XML Support"
Sub Load(ByVal XMLValues As IEnumerable(Of KeyValuePair(Of String, String)))
#End Region
#Region "Initialization"
Sub BeginInit()
Sub EndInit()
@@ -36,6 +41,7 @@ Namespace Plugin
Sub EndEdit()
#End Region
#Region "Site availability"
Property AvailableText As String
Function Available(ByVal What As Download, ByVal Silent As Boolean) As Boolean
Function ReadyToDownload(ByVal What As Download) As Boolean
#End Region
@@ -45,7 +51,10 @@ Namespace Plugin
Sub AfterDownload(ByVal User As Object, ByVal What As Download)
Sub DownloadDone(ByVal What As Download)
#End Region
Sub Update()
Function Clone(ByVal Full As Boolean) As ISiteSettings
Sub Delete()
Overloads Sub Update()
Overloads Sub Update(ByVal Source As ISiteSettings)
Sub Reset()
Sub OpenSettingsForm()
Sub UserOptions(ByRef Options As Object, ByVal OpenForm As Boolean)

View File

@@ -13,7 +13,7 @@ Imports System.Runtime.InteropServices
<Assembly: AssemblyDescription("Plugin provider for SCrawler")>
<Assembly: AssemblyCompany("AndyProgram")>
<Assembly: AssemblyProduct("SCrawler.PluginProvider")>
<Assembly: AssemblyCopyright("Copyright © 2023")>
<Assembly: AssemblyCopyright("Copyright © 2025")>
<Assembly: AssemblyTrademark("AndyProgram")>
<Assembly: ComVisible(False)>
@@ -32,6 +32,6 @@ Imports System.Runtime.InteropServices
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("2023.4.28.0")>
<Assembly: AssemblyFileVersion("2023.4.28.0")>
<Assembly: AssemblyVersion("2025.6.1.0")>
<Assembly: AssemblyFileVersion("2025.6.1.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -11,6 +11,6 @@ Namespace Plugin
Overloads Sub Add(ByVal Message As String)
Overloads Sub Add(ByVal ex As Exception, ByVal Message As String,
Optional ByVal ShowMainMsg As Boolean = False, Optional ByVal ShowErrorMsg As Boolean = False,
Optional ByVal SendInLog As Boolean = True)
Optional ByVal SendToLog As Boolean = True)
End Interface
End Namespace

View File

@@ -11,6 +11,7 @@ Namespace Plugin
Public UserName As String
Public SiteName As String
Public HostKey As String
Public Options As String
Public Exists As Boolean
Public Sub New(ByVal Site As String, ByVal Name As String)
UserName = Name

View File

@@ -0,0 +1,36 @@
' Copyright (C) 2023 Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
' (at your option) any later version.
'
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Namespace Plugin
''' <summary>Represents errors that occur during downloading to be thrown to the root downloading function.</summary>
Public Class ExitException : Inherits Exception
''' <summary>Add only the message to the log, without adding a <see cref="StackTrace"/>. Default: <see langword="True"/>.</summary>
''' <returns><see langword="True"/> if only the message should be added to the log; otherwise the stack trace will also be added.</returns>
Public Property SimpleLogLine As Boolean = True
''' <summary>Don't add a message to the log. Default: <see langword="False"/>.</summary>
''' <returns><see langword="True"/> if the error is exit-only and there is no need to add a message to the log; otherwise add a message to the log.</returns>
Public Property Silent As Boolean = False
''' <summary>Initializes a new instance of the <see cref="ExitException"/> class.</summary>
Public Sub New()
End Sub
''' <summary>Initializes a new instance of the <see cref="ExitException"/> class with a specified error message.</summary>
''' <param name="Message">The message that describes the error.</param>
Public Sub New(ByVal Message As String)
MyBase.New(Message)
End Sub
''' <summary>
''' Initializes a new instance of the <see cref="ExitException"/> class with a specified error message
''' and a reference to the inner exception that is the cause of this exception.
''' </summary>
''' <param name="Message">The error message that explains the reason for the exception.</param>
''' <param name="InnerException">The exception that is the cause of the current exception, or a null reference (Nothing in Visual Basic) if no inner exception is specified.</param>
Public Sub New(ByVal Message As String, ByVal InnerException As Exception)
MyBase.New(Message, InnerException)
End Sub
End Class
End Namespace

View File

@@ -35,6 +35,9 @@ Namespace Plugin
Public Property DownloadState As UserMediaStates Implements IUserMedia.DownloadState
Public Property PostID As String Implements IUserMedia.PostID
Public Property PostDate As Date? Implements IUserMedia.PostDate
Public Property PostText As String Implements IUserMedia.PostText
Public Property PostTextFile As String Implements IUserMedia.PostTextFile
Public Property PostTextFileSpecialFolder As Boolean Implements IUserMedia.PostTextFileSpecialFolder
Public Property SpecialFolder As String Implements IUserMedia.SpecialFolder
Public Property Attempts As Integer Implements IUserMedia.Attempts
Public Property [Object] As Object Implements IUserMedia.Object
@@ -48,6 +51,9 @@ Namespace Plugin
Property DownloadState As UserMediaStates
Property PostID As String
Property PostDate As Date?
Property PostText As String
Property PostTextFile As String
Property PostTextFileSpecialFolder As Boolean
Property SpecialFolder As String
Property Attempts As Integer
Property [Object] As Object

View File

@@ -9,8 +9,24 @@
Namespace Plugin
Public NotInheritable Class PropertyValue : Implements IPropertyValue
Public Event ValueChanged As IPropertyValue.ValueChangedEventHandler Implements IPropertyValue.ValueChanged
Public Event CheckedChanged As IPropertyValue.ValueChangedEventHandler
Public Property [Type] As Type Implements IPropertyValue.Type
Public Property OnChangeFunction As IPropertyValue.ValueChangedEventHandler
Public Property OnCheckboxCheckedChange As EventHandler(Of PropertyValueEventArgs)
Private _Checked As Boolean = False
Public Property Checked As Boolean
Get
Return _Checked
End Get
Set(ByVal IsChecked As Boolean)
_Checked = IsChecked
If Not _Initialization Then
If Not OnCheckboxCheckedChange Is Nothing Then OnCheckboxCheckedChange.Invoke(Me, EventArgs.Empty)
RaiseEvent CheckedChanged(_Checked)
End If
End Set
End Property
Private _Initialization As Boolean = False
''' <inheritdoc cref="PropertyValue.New(Object, Type, ByRef IPropertyValue.ValueChangedEventHandler)"/>
''' <exception cref="ArgumentNullException"></exception>
Public Sub New(ByVal InitValue As Object)
@@ -41,10 +57,26 @@ Namespace Plugin
End Get
Set(ByVal NewValue As Object)
_Value = NewValue
If Not OnChangeFunction Is Nothing Then OnChangeFunction.Invoke(Value)
RaiseEvent ValueChanged(_Value)
If Not _Initialization Then
If Not OnChangeFunction Is Nothing Then OnChangeFunction.Invoke(Value)
RaiseEvent ValueChanged(_Value)
End If
End Set
End Property
Public Sub BeginInit()
_Initialization = True
End Sub
Public Sub EndInit()
_Initialization = False
End Sub
Public Sub Clone(ByVal Source As PropertyValue)
_Initialization = True
Type = Source.Type
OnChangeFunction = Source.OnChangeFunction
_Value = Source._Value
_Checked = Source._Checked
_Initialization = False
End Sub
End Class
Public Interface IPropertyValue
''' <summary>Event for internal exchange</summary>
@@ -55,4 +87,8 @@ Namespace Plugin
''' <summary>Property value</summary>
Property Value As Object
End Interface
Public Class PropertyValueEventArgs : Inherits EventArgs
Public Property Checked As Boolean = False
Public Property ControlEnabled As Boolean = True
End Class
End Namespace

View File

@@ -107,6 +107,7 @@
<Compile Include="Objects\ExchangeOptions.vb" />
<Compile Include="ObjectInterfaces\ILogProvider.vb" />
<Compile Include="Interfaces\IPluginContentProvider.vb" />
<Compile Include="Objects\ExitException.vb" />
<Compile Include="Objects\PluginUserMedia.vb" />
<Compile Include="Interfaces\ISiteSettings.vb" />
<Compile Include="ObjectInterfaces\IThrower.vb" />

View File

@@ -0,0 +1,3 @@
[*.vb]
# Modifier preferences
file_header_template = Copyright (C) Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.8 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 320 B

View File

@@ -0,0 +1,33 @@
' Copyright (C) Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
' (at your option) any later version.
'
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Namespace [Shared]
Public Module Functions
Public Const NewReleaseFolderName As String = "__NewRelease"
Public Function GetCurrentMaxVer(Optional ByVal Path As SFile = Nothing) As Version
Try
If Path.IsEmptyString Then Path = Application.StartupPath.CSFileP
If Path.Exists(SFO.Path, False) Then
Dim versions As New List(Of Version)
Dim v As FileVersionInfo
With SFile.GetFiles(Path, "*.exe",, EDP.ReturnValue).ListIfNothing.Where(Function(f) f.Name = "SCrawler" Or f.Name = "YouTubeDownloader")
If .ListExists Then
For Each f As SFile In .Self
v = FileVersionInfo.GetVersionInfo(f)
versions.Add(New Version(v.ProductVersion))
Next
End If
End With
If versions.Count > 0 Then Return versions.Max
End If
Catch
End Try
Return Nothing
End Function
End Module
End Namespace

View File

@@ -0,0 +1,13 @@
'------------------------------------------------------------------------------
' <auto-generated>
' This code was generated by a tool.
' Runtime Version:4.0.30319.42000
'
' Changes to this file may cause incorrect behavior and will be lost if
' the code is regenerated.
' </auto-generated>
'------------------------------------------------------------------------------
Option Strict On
Option Explicit On

View File

@@ -0,0 +1,10 @@
<?xml version="1.0" encoding="utf-8"?>
<MyApplicationData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<MySubMain>false</MySubMain>
<SingleInstance>false</SingleInstance>
<ShutdownMode>0</ShutdownMode>
<EnableVisualStyles>true</EnableVisualStyles>
<AuthenticationMode>0</AuthenticationMode>
<ApplicationType>1</ApplicationType>
<SaveMySettingsOnExit>true</SaveMySettingsOnExit>
</MyApplicationData>

View File

@@ -0,0 +1,37 @@
Imports System.Resources
Imports System
Imports System.Reflection
Imports System.Runtime.InteropServices
' General Information about an assembly is controlled through the following
' set of attributes. Change these attribute values to modify the information
' associated with an assembly.
' Review the values of the assembly attributes
<Assembly: AssemblyTitle("SCrawler.Shared")>
<Assembly: AssemblyDescription("SCrawler shared functions")>
<Assembly: AssemblyCompany("AndyProgram")>
<Assembly: AssemblyProduct("SCrawler.Shared")>
<Assembly: AssemblyCopyright("Copyright © 2024")>
<Assembly: AssemblyTrademark("AndyProgram")>
<Assembly: ComVisible(False)>
'The following GUID is for the ID of the typelib if this project is exposed to COM
<Assembly: Guid("75a22c7c-6f90-49cb-852b-078ea0b8f2b6")>
' Version information for an assembly consists of the following four values:
'
' Major Version
' Minor Version
' Build Number
' Revision
'
' You can specify all the values or you can default the Build and Revision Numbers
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("2025.6.1.0")>
<Assembly: AssemblyFileVersion("2025.6.1.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -0,0 +1,83 @@
'------------------------------------------------------------------------------
' <auto-generated>
' This code was generated by a tool.
' Runtime Version:4.0.30319.42000
'
' Changes to this file may cause incorrect behavior and will be lost if
' the code is regenerated.
' </auto-generated>
'------------------------------------------------------------------------------
Option Strict On
Option Explicit On
Imports System
Namespace My.Resources
'This class was auto-generated by the StronglyTypedResourceBuilder
'class via a tool like ResGen or Visual Studio.
'To add or remove a member, edit your .ResX file then rerun ResGen
'with the /str option, or rebuild your VS project.
'''<summary>
''' A strongly-typed resource class, for looking up localized strings, etc.
'''</summary>
<Global.System.CodeDom.Compiler.GeneratedCodeAttribute("System.Resources.Tools.StronglyTypedResourceBuilder", "16.0.0.0"), _
Global.System.Diagnostics.DebuggerNonUserCodeAttribute(), _
Global.System.Runtime.CompilerServices.CompilerGeneratedAttribute(), _
Global.Microsoft.VisualBasic.HideModuleNameAttribute()> _
Public Module Resources
Private resourceMan As Global.System.Resources.ResourceManager
Private resourceCulture As Global.System.Globalization.CultureInfo
'''<summary>
''' Returns the cached ResourceManager instance used by this class.
'''</summary>
<Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)> _
Public ReadOnly Property ResourceManager() As Global.System.Resources.ResourceManager
Get
If Object.ReferenceEquals(resourceMan, Nothing) Then
Dim temp As Global.System.Resources.ResourceManager = New Global.System.Resources.ResourceManager("SCrawler.Resources", GetType(Resources).Assembly)
resourceMan = temp
End If
Return resourceMan
End Get
End Property
'''<summary>
''' Overrides the current thread's CurrentUICulture property for all
''' resource lookups using this strongly typed resource class.
'''</summary>
<Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)> _
Public Property Culture() As Global.System.Globalization.CultureInfo
Get
Return resourceCulture
End Get
Set
resourceCulture = value
End Set
End Property
'''<summary>
''' Looks up a localized resource of type System.Drawing.Icon similar to (Icon).
'''</summary>
Public ReadOnly Property FilterIcon() As System.Drawing.Icon
Get
Dim obj As Object = ResourceManager.GetObject("FilterIcon", resourceCulture)
Return CType(obj,System.Drawing.Icon)
End Get
End Property
'''<summary>
''' Looks up a localized resource of type System.Drawing.Bitmap.
'''</summary>
Public ReadOnly Property FilterPic() As System.Drawing.Bitmap
Get
Dim obj As Object = ResourceManager.GetObject("FilterPic", resourceCulture)
Return CType(obj,System.Drawing.Bitmap)
End Get
End Property
End Module
End Namespace

View File

@@ -0,0 +1,127 @@
<?xml version="1.0" encoding="utf-8"?>
<root>
<!--
Microsoft ResX Schema
Version 2.0
The primary goals of this format is to allow a simple XML format
that is mostly human readable. The generation and parsing of the
various data types are done through the TypeConverter classes
associated with the data types.
Example:
... ado.net/XML headers & schema ...
<resheader name="resmimetype">text/microsoft-resx</resheader>
<resheader name="version">2.0</resheader>
<resheader name="reader">System.Resources.ResXResourceReader, System.Windows.Forms, ...</resheader>
<resheader name="writer">System.Resources.ResXResourceWriter, System.Windows.Forms, ...</resheader>
<data name="Name1"><value>this is my long string</value><comment>this is a comment</comment></data>
<data name="Color1" type="System.Drawing.Color, System.Drawing">Blue</data>
<data name="Bitmap1" mimetype="application/x-microsoft.net.object.binary.base64">
<value>[base64 mime encoded serialized .NET Framework object]</value>
</data>
<data name="Icon1" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
<value>[base64 mime encoded string representing a byte array form of the .NET Framework object]</value>
<comment>This is a comment</comment>
</data>
There are any number of "resheader" rows that contain simple
name/value pairs.
Each data row contains a name, and value. The row also contains a
type or mimetype. Type corresponds to a .NET class that support
text/value conversion through the TypeConverter architecture.
Classes that don't support this are serialized and stored with the
mimetype set.
The mimetype is used for serialized objects, and tells the
ResXResourceReader how to depersist the object. This is currently not
extensible. For a given mimetype the value must be set accordingly:
Note - application/x-microsoft.net.object.binary.base64 is the format
that the ResXResourceWriter will generate, however the reader can
read any of the formats listed below.
mimetype: application/x-microsoft.net.object.binary.base64
value : The object must be serialized with
: System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
: and then encoded with base64 encoding.
mimetype: application/x-microsoft.net.object.soap.base64
value : The object must be serialized with
: System.Runtime.Serialization.Formatters.Soap.SoapFormatter
: and then encoded with base64 encoding.
mimetype: application/x-microsoft.net.object.bytearray.base64
value : The object must be serialized into a byte array
: using a System.ComponentModel.TypeConverter
: and then encoded with base64 encoding.
-->
<xsd:schema id="root" xmlns="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
<xsd:import namespace="http://www.w3.org/XML/1998/namespace" />
<xsd:element name="root" msdata:IsDataSet="true">
<xsd:complexType>
<xsd:choice maxOccurs="unbounded">
<xsd:element name="metadata">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" />
</xsd:sequence>
<xsd:attribute name="name" use="required" type="xsd:string" />
<xsd:attribute name="type" type="xsd:string" />
<xsd:attribute name="mimetype" type="xsd:string" />
<xsd:attribute ref="xml:space" />
</xsd:complexType>
</xsd:element>
<xsd:element name="assembly">
<xsd:complexType>
<xsd:attribute name="alias" type="xsd:string" />
<xsd:attribute name="name" type="xsd:string" />
</xsd:complexType>
</xsd:element>
<xsd:element name="data">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
<xsd:element name="comment" type="xsd:string" minOccurs="0" msdata:Ordinal="2" />
</xsd:sequence>
<xsd:attribute name="name" type="xsd:string" use="required" msdata:Ordinal="1" />
<xsd:attribute name="type" type="xsd:string" msdata:Ordinal="3" />
<xsd:attribute name="mimetype" type="xsd:string" msdata:Ordinal="4" />
<xsd:attribute ref="xml:space" />
</xsd:complexType>
</xsd:element>
<xsd:element name="resheader">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
</xsd:sequence>
<xsd:attribute name="name" type="xsd:string" use="required" />
</xsd:complexType>
</xsd:element>
</xsd:choice>
</xsd:complexType>
</xsd:element>
</xsd:schema>
<resheader name="resmimetype">
<value>text/microsoft-resx</value>
</resheader>
<resheader name="version">
<value>2.0</value>
</resheader>
<resheader name="reader">
<value>System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
</resheader>
<resheader name="writer">
<value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
</resheader>
<assembly alias="System.Windows.Forms" name="System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<data name="FilterIcon" type="System.Resources.ResXFileRef, System.Windows.Forms">
<value>..\Content\Icons\FilterIcon.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
</data>
<data name="FilterPic" type="System.Resources.ResXFileRef, System.Windows.Forms">
<value>..\Content\Images\FilterPic.png;System.Drawing.Bitmap, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
</data>
</root>

View File

@@ -0,0 +1,73 @@
'------------------------------------------------------------------------------
' <auto-generated>
' This code was generated by a tool.
' Runtime Version:4.0.30319.42000
'
' Changes to this file may cause incorrect behavior and will be lost if
' the code is regenerated.
' </auto-generated>
'------------------------------------------------------------------------------
Option Strict On
Option Explicit On
Namespace My
<Global.System.Runtime.CompilerServices.CompilerGeneratedAttribute(), _
Global.System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.VisualStudio.Editors.SettingsDesigner.SettingsSingleFileGenerator", "16.10.0.0"), _
Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)> _
Partial Friend NotInheritable Class MySettings
Inherits Global.System.Configuration.ApplicationSettingsBase
Private Shared defaultInstance As MySettings = CType(Global.System.Configuration.ApplicationSettingsBase.Synchronized(New MySettings()),MySettings)
#Region "My.Settings Auto-Save Functionality"
#If _MyType = "WindowsForms" Then
Private Shared addedHandler As Boolean
Private Shared addedHandlerLockObject As New Object
<Global.System.Diagnostics.DebuggerNonUserCodeAttribute(), Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)> _
Private Shared Sub AutoSaveSettings(sender As Global.System.Object, e As Global.System.EventArgs)
If My.Application.SaveMySettingsOnExit Then
My.Settings.Save()
End If
End Sub
#End If
#End Region
Public Shared ReadOnly Property [Default]() As MySettings
Get
#If _MyType = "WindowsForms" Then
If Not addedHandler Then
SyncLock addedHandlerLockObject
If Not addedHandler Then
AddHandler My.Application.Shutdown, AddressOf AutoSaveSettings
addedHandler = True
End If
End SyncLock
End If
#End If
Return defaultInstance
End Get
End Property
End Class
End Namespace
Namespace My
<Global.Microsoft.VisualBasic.HideModuleNameAttribute(), _
Global.System.Diagnostics.DebuggerNonUserCodeAttribute(), _
Global.System.Runtime.CompilerServices.CompilerGeneratedAttribute()> _
Friend Module MySettingsProperty
<Global.System.ComponentModel.Design.HelpKeywordAttribute("My.Settings")> _
Friend ReadOnly Property Settings() As Global.SCrawler.My.MySettings
Get
Return Global.SCrawler.My.MySettings.Default
End Get
End Property
End Module
End Namespace

View File

@@ -0,0 +1,7 @@
<?xml version='1.0' encoding='utf-8'?>
<SettingsFile xmlns="http://schemas.microsoft.com/VisualStudio/2004/01/settings" CurrentProfile="(Default)" UseMySettingsClassName="true">
<Profiles>
<Profile Name="(Default)" />
</Profiles>
<Settings />
</SettingsFile>

View File

@@ -0,0 +1,156 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
<ProjectGuid>{DC634700-24C7-42DD-BF8F-87E6CC54E625}</ProjectGuid>
<OutputType>Library</OutputType>
<RootNamespace>SCrawler</RootNamespace>
<AssemblyName>SCrawler.Shared</AssemblyName>
<FileAlignment>512</FileAlignment>
<MyType>Windows</MyType>
<TargetFrameworkVersion>v4.6.1</TargetFrameworkVersion>
<Deterministic>true</Deterministic>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<DebugSymbols>true</DebugSymbols>
<DebugType>full</DebugType>
<DefineDebug>true</DefineDebug>
<DefineTrace>true</DefineTrace>
<OutputPath>bin\Debug\</OutputPath>
<DocumentationFile>
</DocumentationFile>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
<DebugType>pdbonly</DebugType>
<DefineDebug>false</DefineDebug>
<DefineTrace>true</DefineTrace>
<Optimize>true</Optimize>
<OutputPath>bin\Release\</OutputPath>
<DocumentationFile>
</DocumentationFile>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
</PropertyGroup>
<PropertyGroup>
<OptionExplicit>On</OptionExplicit>
</PropertyGroup>
<PropertyGroup>
<OptionCompare>Binary</OptionCompare>
</PropertyGroup>
<PropertyGroup>
<OptionStrict>Off</OptionStrict>
</PropertyGroup>
<PropertyGroup>
<OptionInfer>On</OptionInfer>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|x64'">
<DebugSymbols>true</DebugSymbols>
<DefineDebug>true</DefineDebug>
<DefineTrace>true</DefineTrace>
<OutputPath>bin\x64\Debug\</OutputPath>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<DebugType>full</DebugType>
<PlatformTarget>x64</PlatformTarget>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|x64'">
<DefineTrace>true</DefineTrace>
<OutputPath>bin\x64\Release\</OutputPath>
<Optimize>true</Optimize>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<DebugType>pdbonly</DebugType>
<PlatformTarget>x64</PlatformTarget>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|x86'">
<DebugSymbols>true</DebugSymbols>
<DefineDebug>true</DefineDebug>
<DefineTrace>true</DefineTrace>
<OutputPath>bin\x86\Debug\</OutputPath>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<DebugType>full</DebugType>
<PlatformTarget>x86</PlatformTarget>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|x86'">
<DefineTrace>true</DefineTrace>
<OutputPath>bin\x86\Release\</OutputPath>
<Optimize>true</Optimize>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<DebugType>pdbonly</DebugType>
<PlatformTarget>x86</PlatformTarget>
</PropertyGroup>
<ItemGroup>
<Reference Include="System" />
<Reference Include="System.Data" />
<Reference Include="System.Drawing" />
<Reference Include="System.Windows.Forms" />
<Reference Include="System.Xml" />
<Reference Include="System.Core" />
<Reference Include="System.Xml.Linq" />
<Reference Include="System.Data.DataSetExtensions" />
<Reference Include="System.Net.Http" />
</ItemGroup>
<ItemGroup>
<Import Include="Microsoft.VisualBasic" />
<Import Include="PersonalUtilities.Functions" />
<Import Include="System" />
<Import Include="System.Collections" />
<Import Include="System.Collections.Generic" />
<Import Include="System.Data" />
<Import Include="System.Diagnostics" />
<Import Include="System.Linq" />
<Import Include="System.Windows.Forms" />
<Import Include="System.Xml.Linq" />
<Import Include="System.Threading.Tasks" />
</ItemGroup>
<ItemGroup>
<Compile Include="Functions.vb" />
<Compile Include="My Project\AssemblyInfo.vb" />
<Compile Include="My Project\Application.Designer.vb">
<AutoGen>True</AutoGen>
<DependentUpon>Application.myapp</DependentUpon>
<DesignTime>True</DesignTime>
</Compile>
<Compile Include="My Project\Resources.Designer.vb">
<AutoGen>True</AutoGen>
<DesignTime>True</DesignTime>
<DependentUpon>Resources.resx</DependentUpon>
</Compile>
<Compile Include="My Project\Settings.Designer.vb">
<AutoGen>True</AutoGen>
<DependentUpon>Settings.settings</DependentUpon>
<DesignTimeSharedInput>True</DesignTimeSharedInput>
</Compile>
</ItemGroup>
<ItemGroup>
<EmbeddedResource Include="My Project\Resources.resx">
<Generator>PublicVbMyResourcesResXFileCodeGenerator</Generator>
<LastGenOutput>Resources.Designer.vb</LastGenOutput>
<CustomToolNamespace>My.Resources</CustomToolNamespace>
<SubType>Designer</SubType>
</EmbeddedResource>
</ItemGroup>
<ItemGroup>
<None Include=".editorconfig" />
<None Include="My Project\Application.myapp">
<Generator>MyApplicationCodeGenerator</Generator>
<LastGenOutput>Application.Designer.vb</LastGenOutput>
</None>
<None Include="My Project\Settings.settings">
<Generator>SettingsSingleFileGenerator</Generator>
<CustomToolNamespace>My</CustomToolNamespace>
<LastGenOutput>Settings.Designer.vb</LastGenOutput>
</None>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\MyUtilities\PersonalUtilities\PersonalUtilities.vbproj">
<Project>{8405896b-2685-4916-bc93-1fb514c323a9}</Project>
<Name>PersonalUtilities</Name>
</ProjectReference>
</ItemGroup>
<ItemGroup>
<Content Include="Content\Icons\FilterIcon.ico" />
<Content Include="Content\Images\FilterPic.png" />
</ItemGroup>
<Import Project="$(MSBuildToolsPath)\Microsoft.VisualBasic.targets" />
</Project>

View File

@@ -0,0 +1,3 @@
[*.vb]
# Modifier preferences
file_header_template = Copyright (C) Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>

View File

@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1" />
</startup>
</configuration>

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.4 KiB

229
SCrawler.Updater/MainMod.vb Normal file
View File

@@ -0,0 +1,229 @@
' Copyright (C) Andy https://github.com/AAndyProgram
' This program is free software: you can redistribute it and/or modify
' it under the terms of the GNU General Public License as published by
' the Free Software Foundation, either version 3 of the License, or
' (at your option) any later version.
'
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Imports System.Net
Imports System.IO.Compression
Imports PersonalUtilities.Functions
Imports PersonalUtilities.Functions.XML
Imports PersonalUtilities.Tools.Web.Clients
Imports PersonalUtilities.Tools.Web.Documents.JSON
Imports SCrawler.Shared
Public Module MainMod
Private MyProcessID As Integer = -1
Private MyWorkingPath As SFile = Nothing
Private ReadOnly ProcessNames As String() = {"SCrawler", "YouTubeDownloader", "Updater"}
Private Silent As Boolean = False
Private Function GetConsoleResponse(ByVal Request As String) As String
Console.Write(Request)
Return Console.ReadLine
End Function
Private Function DownloadFile(ByVal URL As String, ByVal Destination As SFile) As Boolean
Try
Dim lastPerc% = -1
Dim currentCursor% = Console.CursorTop
Using w As New RWebClient With {.AsyncMode = True}
AddHandler w.DownloadProgressChanged,
New DownloadProgressChangedEventHandler(Sub(ByVal Sender As Object, ByVal e As DownloadProgressChangedEventArgs)
If lastPerc < e.ProgressPercentage Then
lastPerc = e.ProgressPercentage
Console.SetCursorPosition(0, currentCursor)
Console.Write("{0}% completed", e.ProgressPercentage)
End If
End Sub)
Return w.DownloadFile(URL, Destination, EDP.ReturnValue)
End Using
Catch ex As Exception
Return False
End Try
End Function
Public Sub Main()
Try
MyWorkingPath = AppDomain.CurrentDomain.BaseDirectory.CSFileP
MyProcessID = Process.GetCurrentProcess.Id
Console.Title = "SCrawler updater"
With Environment.GetCommandLineArgs
If .ListExists(2) Then Silent = .Self()(1).FromXML(Of Boolean)(False)
End With
Dim currentDir As SFile = MyWorkingPath.CutPath
Dim extractionDir As SFile = $"{currentDir.CSFilePS}{NewReleaseFolderName}\"
If extractionDir.Exists(SFO.Path, False) Then extractionDir.Delete(SFO.Path, SFODelete.DeletePermanently, EDP.None)
Dim currVer As Version = GetCurrentMaxVer(currentDir)
If currVer Is Nothing Then
Console.WriteLine("The current version of the program cannot be determined")
Else
Console.WriteLine($"The current version is {currVer} (x{IIf(Environment.Is64BitProcess, 64, 86)})")
Dim release As GitRelease = GetGitRelease()
If Not release.URL.IsEmptyString And Not release.Version Is Nothing Then
If release.Version > currVer Then
Console.WriteLine($"The new version is {release.Version} ({release.Name})")
If Not Silent AndAlso GetConsoleResponse("Do you want to update the program? (y/n): ").IfNullOrEmpty("n") = "n" Then Exit Sub
If ActiveProcessesExist() Then
Console.WriteLine("One of the SCrawler programs is still running. Waiting for all SCrawler programs to close.")
While ActiveProcessesExist() : Threading.Thread.Sleep(100) : End While
Console.WriteLine("All SCrawler programs are closed.")
End If
If extractionDir.Exists(SFO.Path, True) Then
Dim destFile As SFile = $"{extractionDir.CSFilePS}{New SFile(release.URL).File}"
Console.WriteLine("Downloading new version...")
If DownloadFile(release.URL, destFile) Then
Console.WriteLine("")
Console.WriteLine("New version downloaded!")
Console.WriteLine("Extracting files...")
ZipFile.ExtractToDirectory(destFile, extractionDir)
Console.WriteLine("Files extracted!")
destFile.Delete(SFO.File, SFODelete.DeletePermanently, EDP.None)
If Not MoveFiles(extractionDir, currentDir) Then GetConsoleResponse("Unable to update the program. Press Enter to exit") : Exit Sub
Else
extractionDir.Delete(SFO.Path, SFODelete.DeletePermanently, EDP.None)
Console.WriteLine("Unable to download new version")
End If
Else
Console.WriteLine("Unable to create temp directory")
End If
Else
Console.WriteLine("The program is up to date")
End If
Else
Console.WriteLine("Unable to get information about new version")
End If
End If
Catch ex As Exception
Console.WriteLine("An error occurred during update")
Console.WriteLine(ex.Message)
Finally
GetConsoleResponse("Press Enter to exit")
End Try
End Sub
Private Function MoveFiles(ByVal Source As SFile, ByVal Destination As SFile) As Boolean
Console.WriteLine("Updating files")
Try
Dim oldFiles As List(Of SFile) = SFile.GetFiles(Destination,,, EDP.ReturnValue)
Dim oldFolders As List(Of SFile) = SFile.GetDirectories(Destination,,, EDP.ReturnValue)
Dim newFiles As List(Of SFile) = SFile.GetFiles(Source,,, EDP.ReturnValue)
Dim newFolders As List(Of SFile) = SFile.GetDirectories(Source,,, EDP.ReturnValue)
Dim obj As SFile = Nothing
Dim wSegment As String = MyWorkingPath.Segments.Last
Dim filesPredicate As Predicate(Of SFile) = Function(ByVal f As SFile) As Boolean
If obj = f Or obj.Name = f.Name Then
f.Delete(SFO.File, SFODelete.DeleteToRecycleBin, EDP.None)
Return True
Else
Return False
End If
End Function
Dim foldersPredicate As Predicate(Of SFile) = Function(ByVal f As SFile) As Boolean
Dim ls$ = f.Segments.Last
If ls = obj.Segments.Last And Not ls = NewReleaseFolderName And Not ls = wSegment Then
f.Delete(SFO.Path, SFODelete.DeleteToRecycleBin, EDP.None)
Return True
Else
Return False
End If
End Function
Dim getDestFile As Func(Of SFile, Boolean, SFile) = Function(ByVal f As SFile, ByVal isFolder As Boolean) As SFile
Dim ff As SFile = f
If isFolder Then
ff = $"{Destination.PathWithSeparator}{f.Segments.Last}\"
Else
ff.Path = Destination.Path
End If
Console.WriteLine(ff)
Return ff
End Function
If newFiles.ListExists Then
If oldFiles.ListExists Then
For Each obj In newFiles : oldFiles.RemoveAll(filesPredicate) : Next
End If
newFiles.ForEach(Sub(ff) SFile.Move(ff, getDestFile(ff, False), SFO.File, True, SFODelete.DeleteToRecycleBin, EDP.None))
End If
If newFolders.ListExists Then
If oldFolders.ListExists Then
For Each obj In newFolders : oldFolders.RemoveAll(foldersPredicate) : Next
End If
newFolders.ForEach(Sub(ff) If Not ff.Segments.Last = wSegment Then _
SFile.Move(ff, getDestFile(ff, True), SFO.Path, True, SFODelete.DeleteToRecycleBin, EDP.None))
End If
Console.WriteLine("Files updated")
Return True
Catch
Return False
End Try
End Function
Private Function ActiveProcessesExist() As Boolean
Try
Return Process.GetProcesses.Any(Function(p) ProcessNames.Contains(p.ProcessName) And Not p.Id = MyProcessID)
Catch
Return True
End Try
End Function
Private Structure GitRelease
Friend URL As String
Friend Name As String
Friend Version As Version
End Structure
Private Function GetGitRelease() As GitRelease
Try
Dim nameEnd$ = $"_x{IIf(Environment.Is64BitProcess, 64, 86)}.zip"
Dim name$, relName$, relTag$
Using resp As New Responser With {.Accept = "application/vnd.github.v3+json"}
Dim r$ = resp.GetResponse("https://api.github.com/repos/AAndyProgram/SCrawler/releases",, EDP.ReturnValue)
If Not r.IsEmptyString Then
Dim getver As Func(Of String, Version) = Function(ByVal input As String) As Version
Try
If Not input.IsEmptyString Then
If input.ToLower.StartsWith("scrawler") Then
Return New Version(input.Split("_")(1))
Else
Return New Version(input)
End If
End If
Catch
End Try
Return Nothing
End Function
Using j As EContainer = JsonDocument.Parse(r, EDP.ReturnValue)
If j.ListExists Then
With j.FirstOrDefault(Function(e) Not e.Value("draft").FromXML(Of Boolean) And Not e.Value("prerelease").FromXML(Of Boolean))
If .ListExists Then
relName = .Value("name")
relTag = .Value("tag_name")
With .Item("assets")
If .ListExists Then
For Each asset As EContainer In .Self
name = asset.Value("name")
If Not name.IsEmptyString AndAlso name.EndsWith(nameEnd) Then _
Return New GitRelease With {
.Name = name,
.URL = asset.Value("browser_download_url"),
.Version = getver(name).IfNullOrEmpty(getver(relName).IfNullOrEmpty(getver(relTag)))}
Next
End If
End With
End If
End With
End If
End Using
End If
End Using
Catch
End Try
Return Nothing
End Function
End Module

View File

@@ -0,0 +1,13 @@
'------------------------------------------------------------------------------
' <auto-generated>
' This code was generated by a tool.
' Runtime Version:4.0.30319.42000
'
' Changes to this file may cause incorrect behavior and will be lost if
' the code is regenerated.
' </auto-generated>
'------------------------------------------------------------------------------
Option Strict On
Option Explicit On

View File

@@ -0,0 +1,10 @@
<?xml version="1.0" encoding="utf-8"?>
<MyApplicationData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<MySubMain>false</MySubMain>
<SingleInstance>false</SingleInstance>
<ShutdownMode>0</ShutdownMode>
<EnableVisualStyles>true</EnableVisualStyles>
<AuthenticationMode>0</AuthenticationMode>
<ApplicationType>2</ApplicationType>
<SaveMySettingsOnExit>true</SaveMySettingsOnExit>
</MyApplicationData>

View File

@@ -0,0 +1,37 @@
Imports System.Resources
Imports System
Imports System.Reflection
Imports System.Runtime.InteropServices
' General Information about an assembly is controlled through the following
' set of attributes. Change these attribute values to modify the information
' associated with an assembly.
' Review the values of the assembly attributes
<Assembly: AssemblyTitle("SCrawler updater")>
<Assembly: AssemblyDescription("SCrawler updater")>
<Assembly: AssemblyCompany("AndyProgram")>
<Assembly: AssemblyProduct("SCrawler.Updater")>
<Assembly: AssemblyCopyright("Copyright © 2024")>
<Assembly: AssemblyTrademark("AndyProgram")>
<Assembly: ComVisible(False)>
'The following GUID is for the ID of the typelib if this project is exposed to COM
<Assembly: Guid("df008b29-ad5e-4271-acfe-650396d15c40")>
' Version information for an assembly consists of the following four values:
'
' Major Version
' Minor Version
' Build Number
' Revision
'
' You can specify all the values or you can default the Build and Revision Numbers
' by using the '*' as shown below:
' <Assembly: AssemblyVersion("1.0.*")>
<Assembly: AssemblyVersion("2023.12.7.0")>
<Assembly: AssemblyFileVersion("2023.12.7.0")>
<Assembly: NeutralResourcesLanguage("en")>

View File

@@ -0,0 +1,73 @@
'------------------------------------------------------------------------------
' <auto-generated>
' This code was generated by a tool.
' Runtime Version:4.0.30319.42000
'
' Changes to this file may cause incorrect behavior and will be lost if
' the code is regenerated.
' </auto-generated>
'------------------------------------------------------------------------------
Option Strict On
Option Explicit On
Imports System
Namespace My.Resources
'This class was auto-generated by the StronglyTypedResourceBuilder
'class via a tool like ResGen or Visual Studio.
'To add or remove a member, edit your .ResX file then rerun ResGen
'with the /str option, or rebuild your VS project.
'''<summary>
''' A strongly-typed resource class, for looking up localized strings, etc.
'''</summary>
<Global.System.CodeDom.Compiler.GeneratedCodeAttribute("System.Resources.Tools.StronglyTypedResourceBuilder", "16.0.0.0"), _
Global.System.Diagnostics.DebuggerNonUserCodeAttribute(), _
Global.System.Runtime.CompilerServices.CompilerGeneratedAttribute(), _
Global.Microsoft.VisualBasic.HideModuleNameAttribute()> _
Friend Module Resources
Private resourceMan As Global.System.Resources.ResourceManager
Private resourceCulture As Global.System.Globalization.CultureInfo
'''<summary>
''' Returns the cached ResourceManager instance used by this class.
'''</summary>
<Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)> _
Friend ReadOnly Property ResourceManager() As Global.System.Resources.ResourceManager
Get
If Object.ReferenceEquals(resourceMan, Nothing) Then
Dim temp As Global.System.Resources.ResourceManager = New Global.System.Resources.ResourceManager("SCrawler.Resources", GetType(Resources).Assembly)
resourceMan = temp
End If
Return resourceMan
End Get
End Property
'''<summary>
''' Overrides the current thread's CurrentUICulture property for all
''' resource lookups using this strongly typed resource class.
'''</summary>
<Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)> _
Friend Property Culture() As Global.System.Globalization.CultureInfo
Get
Return resourceCulture
End Get
Set
resourceCulture = value
End Set
End Property
'''<summary>
''' Looks up a localized resource of type System.Drawing.Icon similar to (Icon).
'''</summary>
Friend ReadOnly Property RainbowIcon_48() As System.Drawing.Icon
Get
Dim obj As Object = ResourceManager.GetObject("RainbowIcon_48", resourceCulture)
Return CType(obj,System.Drawing.Icon)
End Get
End Property
End Module
End Namespace

View File

@@ -0,0 +1,124 @@
<?xml version="1.0" encoding="utf-8"?>
<root>
<!--
Microsoft ResX Schema
Version 2.0
The primary goals of this format is to allow a simple XML format
that is mostly human readable. The generation and parsing of the
various data types are done through the TypeConverter classes
associated with the data types.
Example:
... ado.net/XML headers & schema ...
<resheader name="resmimetype">text/microsoft-resx</resheader>
<resheader name="version">2.0</resheader>
<resheader name="reader">System.Resources.ResXResourceReader, System.Windows.Forms, ...</resheader>
<resheader name="writer">System.Resources.ResXResourceWriter, System.Windows.Forms, ...</resheader>
<data name="Name1"><value>this is my long string</value><comment>this is a comment</comment></data>
<data name="Color1" type="System.Drawing.Color, System.Drawing">Blue</data>
<data name="Bitmap1" mimetype="application/x-microsoft.net.object.binary.base64">
<value>[base64 mime encoded serialized .NET Framework object]</value>
</data>
<data name="Icon1" type="System.Drawing.Icon, System.Drawing" mimetype="application/x-microsoft.net.object.bytearray.base64">
<value>[base64 mime encoded string representing a byte array form of the .NET Framework object]</value>
<comment>This is a comment</comment>
</data>
There are any number of "resheader" rows that contain simple
name/value pairs.
Each data row contains a name, and value. The row also contains a
type or mimetype. Type corresponds to a .NET class that support
text/value conversion through the TypeConverter architecture.
Classes that don't support this are serialized and stored with the
mimetype set.
The mimetype is used for serialized objects, and tells the
ResXResourceReader how to depersist the object. This is currently not
extensible. For a given mimetype the value must be set accordingly:
Note - application/x-microsoft.net.object.binary.base64 is the format
that the ResXResourceWriter will generate, however the reader can
read any of the formats listed below.
mimetype: application/x-microsoft.net.object.binary.base64
value : The object must be serialized with
: System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
: and then encoded with base64 encoding.
mimetype: application/x-microsoft.net.object.soap.base64
value : The object must be serialized with
: System.Runtime.Serialization.Formatters.Soap.SoapFormatter
: and then encoded with base64 encoding.
mimetype: application/x-microsoft.net.object.bytearray.base64
value : The object must be serialized into a byte array
: using a System.ComponentModel.TypeConverter
: and then encoded with base64 encoding.
-->
<xsd:schema id="root" xmlns="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
<xsd:import namespace="http://www.w3.org/XML/1998/namespace" />
<xsd:element name="root" msdata:IsDataSet="true">
<xsd:complexType>
<xsd:choice maxOccurs="unbounded">
<xsd:element name="metadata">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" />
</xsd:sequence>
<xsd:attribute name="name" use="required" type="xsd:string" />
<xsd:attribute name="type" type="xsd:string" />
<xsd:attribute name="mimetype" type="xsd:string" />
<xsd:attribute ref="xml:space" />
</xsd:complexType>
</xsd:element>
<xsd:element name="assembly">
<xsd:complexType>
<xsd:attribute name="alias" type="xsd:string" />
<xsd:attribute name="name" type="xsd:string" />
</xsd:complexType>
</xsd:element>
<xsd:element name="data">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
<xsd:element name="comment" type="xsd:string" minOccurs="0" msdata:Ordinal="2" />
</xsd:sequence>
<xsd:attribute name="name" type="xsd:string" use="required" msdata:Ordinal="1" />
<xsd:attribute name="type" type="xsd:string" msdata:Ordinal="3" />
<xsd:attribute name="mimetype" type="xsd:string" msdata:Ordinal="4" />
<xsd:attribute ref="xml:space" />
</xsd:complexType>
</xsd:element>
<xsd:element name="resheader">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
</xsd:sequence>
<xsd:attribute name="name" type="xsd:string" use="required" />
</xsd:complexType>
</xsd:element>
</xsd:choice>
</xsd:complexType>
</xsd:element>
</xsd:schema>
<resheader name="resmimetype">
<value>text/microsoft-resx</value>
</resheader>
<resheader name="version">
<value>2.0</value>
</resheader>
<resheader name="reader">
<value>System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
</resheader>
<resheader name="writer">
<value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
</resheader>
<assembly alias="System.Windows.Forms" name="System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
<data name="RainbowIcon_48" type="System.Resources.ResXFileRef, System.Windows.Forms">
<value>..\Content\Icons\RainbowIcon_48.ico;System.Drawing.Icon, System.Drawing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a</value>
</data>
</root>

View File

@@ -0,0 +1,73 @@
'------------------------------------------------------------------------------
' <auto-generated>
' This code was generated by a tool.
' Runtime Version:4.0.30319.42000
'
' Changes to this file may cause incorrect behavior and will be lost if
' the code is regenerated.
' </auto-generated>
'------------------------------------------------------------------------------
Option Strict On
Option Explicit On
Namespace My
<Global.System.Runtime.CompilerServices.CompilerGeneratedAttribute(), _
Global.System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.VisualStudio.Editors.SettingsDesigner.SettingsSingleFileGenerator", "16.10.0.0"), _
Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)> _
Partial Friend NotInheritable Class MySettings
Inherits Global.System.Configuration.ApplicationSettingsBase
Private Shared defaultInstance As MySettings = CType(Global.System.Configuration.ApplicationSettingsBase.Synchronized(New MySettings()),MySettings)
#Region "My.Settings Auto-Save Functionality"
#If _MyType = "WindowsForms" Then
Private Shared addedHandler As Boolean
Private Shared addedHandlerLockObject As New Object
<Global.System.Diagnostics.DebuggerNonUserCodeAttribute(), Global.System.ComponentModel.EditorBrowsableAttribute(Global.System.ComponentModel.EditorBrowsableState.Advanced)> _
Private Shared Sub AutoSaveSettings(sender As Global.System.Object, e As Global.System.EventArgs)
If My.Application.SaveMySettingsOnExit Then
My.Settings.Save()
End If
End Sub
#End If
#End Region
Public Shared ReadOnly Property [Default]() As MySettings
Get
#If _MyType = "WindowsForms" Then
If Not addedHandler Then
SyncLock addedHandlerLockObject
If Not addedHandler Then
AddHandler My.Application.Shutdown, AddressOf AutoSaveSettings
addedHandler = True
End If
End SyncLock
End If
#End If
Return defaultInstance
End Get
End Property
End Class
End Namespace
Namespace My
<Global.Microsoft.VisualBasic.HideModuleNameAttribute(), _
Global.System.Diagnostics.DebuggerNonUserCodeAttribute(), _
Global.System.Runtime.CompilerServices.CompilerGeneratedAttribute()> _
Friend Module MySettingsProperty
<Global.System.ComponentModel.Design.HelpKeywordAttribute("My.Settings")> _
Friend ReadOnly Property Settings() As Global.SCrawler.My.MySettings
Get
Return Global.SCrawler.My.MySettings.Default
End Get
End Property
End Module
End Namespace

View File

@@ -0,0 +1,7 @@
<?xml version='1.0' encoding='utf-8'?>
<SettingsFile xmlns="http://schemas.microsoft.com/VisualStudio/2004/01/settings" CurrentProfile="(Default)" UseMySettingsClassName="true">
<Profiles>
<Profile Name="(Default)" />
</Profiles>
<Settings />
</SettingsFile>

View File

@@ -0,0 +1,79 @@
<?xml version="1.0" encoding="utf-8"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
<assemblyIdentity version="1.0.0.0" name="MyApplication.app"/>
<trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
<security>
<requestedPrivileges xmlns="urn:schemas-microsoft-com:asm.v3">
<!-- UAC Manifest Options
If you want to change the Windows User Account Control level replace the
requestedExecutionLevel node with one of the following.
<requestedExecutionLevel level="asInvoker" uiAccess="false" />
<requestedExecutionLevel level="requireAdministrator" uiAccess="false" />
<requestedExecutionLevel level="highestAvailable" uiAccess="false" />
Specifying requestedExecutionLevel element will disable file and registry virtualization.
Remove this element if your application requires this virtualization for backwards
compatibility.
-->
<requestedExecutionLevel level="asInvoker" uiAccess="false" />
</requestedPrivileges>
</security>
</trustInfo>
<compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
<application>
<!-- A list of the Windows versions that this application has been tested on
and is designed to work with. Uncomment the appropriate elements
and Windows will automatically select the most compatible environment. -->
<!-- Windows Vista -->
<!--<supportedOS Id="{e2011457-1546-43c5-a5fe-008deee3d3f0}" />-->
<!-- Windows 7 -->
<!--<supportedOS Id="{35138b9a-5d96-4fbd-8e2d-a2440225f93a}" />-->
<!-- Windows 8 -->
<!--<supportedOS Id="{4a2f28e3-53b9-4441-ba9c-d69d4a4a6e38}" />-->
<!-- Windows 8.1 -->
<!--<supportedOS Id="{1f676c76-80e1-4239-95bb-83d0f6d0da78}" />-->
<!-- Windows 10 -->
<!--<supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}" />-->
</application>
</compatibility>
<!-- Indicates that the application is DPI-aware and will not be automatically scaled by Windows at higher
DPIs. Windows Presentation Foundation (WPF) applications are automatically DPI-aware and do not need
to opt in. Windows Forms applications targeting .NET Framework 4.6 that opt into this setting, should
also set the 'EnableWindowsFormsHighDpiAutoResizing' setting to 'true' in their app.config.
Makes the application long-path aware. See https://docs.microsoft.com/windows/win32/fileio/maximum-file-path-limitation -->
<!--
<application xmlns="urn:schemas-microsoft-com:asm.v3">
<windowsSettings>
<dpiAware xmlns="http://schemas.microsoft.com/SMI/2005/WindowsSettings">true</dpiAware>
<longPathAware xmlns="http://schemas.microsoft.com/SMI/2016/WindowsSettings">true</longPathAware>
</windowsSettings>
</application>
-->
<!-- Enable themes for Windows common controls and dialogs (Windows XP and later) -->
<!--
<dependency>
<dependentAssembly>
<assemblyIdentity
type="win32"
name="Microsoft.Windows.Common-Controls"
version="6.0.0.0"
processorArchitecture="*"
publicKeyToken="6595b64144ccf1df"
language="*"
/>
</dependentAssembly>
</dependency>
-->
</assembly>

View File

@@ -0,0 +1,173 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
<ProjectGuid>{71263EEE-E25F-44DD-B0A9-F09047C0BEEA}</ProjectGuid>
<OutputType>Exe</OutputType>
<StartupObject>Sub Main</StartupObject>
<RootNamespace>SCrawler</RootNamespace>
<AssemblyName>Updater</AssemblyName>
<FileAlignment>512</FileAlignment>
<MyType>Console</MyType>
<TargetFrameworkVersion>v4.6.1</TargetFrameworkVersion>
<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
<Deterministic>true</Deterministic>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugSymbols>true</DebugSymbols>
<DebugType>full</DebugType>
<DefineDebug>true</DefineDebug>
<DefineTrace>true</DefineTrace>
<OutputPath>bin\Debug\Updater\</OutputPath>
<DocumentationFile>
</DocumentationFile>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<Prefer32Bit>false</Prefer32Bit>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugType>pdbonly</DebugType>
<DefineDebug>false</DefineDebug>
<DefineTrace>true</DefineTrace>
<Optimize>true</Optimize>
<OutputPath>bin\Release\Updater\</OutputPath>
<DocumentationFile>
</DocumentationFile>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<Prefer32Bit>false</Prefer32Bit>
</PropertyGroup>
<PropertyGroup>
<OptionExplicit>On</OptionExplicit>
</PropertyGroup>
<PropertyGroup>
<OptionCompare>Binary</OptionCompare>
</PropertyGroup>
<PropertyGroup>
<OptionStrict>Off</OptionStrict>
</PropertyGroup>
<PropertyGroup>
<OptionInfer>On</OptionInfer>
</PropertyGroup>
<PropertyGroup>
<ApplicationIcon>Content\Icons\RainbowIcon_48.ico</ApplicationIcon>
</PropertyGroup>
<PropertyGroup>
<ApplicationManifest>My Project\app.manifest</ApplicationManifest>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|x64'">
<DebugSymbols>true</DebugSymbols>
<DefineDebug>true</DefineDebug>
<DefineTrace>true</DefineTrace>
<OutputPath>bin\x64\Debug\Updater\</OutputPath>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<DebugType>full</DebugType>
<PlatformTarget>x64</PlatformTarget>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|x64'">
<DefineTrace>true</DefineTrace>
<OutputPath>bin\x64\Release\Updater\</OutputPath>
<Optimize>true</Optimize>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<DebugType>pdbonly</DebugType>
<PlatformTarget>x64</PlatformTarget>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|x86'">
<DebugSymbols>true</DebugSymbols>
<DefineDebug>true</DefineDebug>
<DefineTrace>true</DefineTrace>
<OutputPath>bin\x86\Debug\Updater\</OutputPath>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<DebugType>full</DebugType>
<PlatformTarget>x86</PlatformTarget>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|x86'">
<DefineTrace>true</DefineTrace>
<OutputPath>bin\x86\Release\Updater\</OutputPath>
<Optimize>true</Optimize>
<NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
<DebugType>pdbonly</DebugType>
<PlatformTarget>x86</PlatformTarget>
</PropertyGroup>
<ItemGroup>
<Reference Include="System" />
<Reference Include="System.Data" />
<Reference Include="System.Deployment" />
<Reference Include="System.Drawing" />
<Reference Include="System.IO.Compression" />
<Reference Include="System.IO.Compression.FileSystem" />
<Reference Include="System.Xml" />
<Reference Include="System.Core" />
<Reference Include="System.Xml.Linq" />
<Reference Include="System.Data.DataSetExtensions" />
<Reference Include="System.Net.Http" />
</ItemGroup>
<ItemGroup>
<Import Include="Microsoft.VisualBasic" />
<Import Include="System" />
<Import Include="System.Collections" />
<Import Include="System.Collections.Generic" />
<Import Include="System.Data" />
<Import Include="System.Diagnostics" />
<Import Include="System.Linq" />
<Import Include="System.Xml.Linq" />
<Import Include="System.Threading.Tasks" />
</ItemGroup>
<ItemGroup>
<Compile Include="MainMod.vb" />
<Compile Include="My Project\AssemblyInfo.vb" />
<Compile Include="My Project\Application.Designer.vb">
<AutoGen>True</AutoGen>
<DependentUpon>Application.myapp</DependentUpon>
<DesignTime>True</DesignTime>
</Compile>
<Compile Include="My Project\Resources.Designer.vb">
<AutoGen>True</AutoGen>
<DesignTime>True</DesignTime>
<DependentUpon>Resources.resx</DependentUpon>
</Compile>
<Compile Include="My Project\Settings.Designer.vb">
<AutoGen>True</AutoGen>
<DependentUpon>Settings.settings</DependentUpon>
<DesignTimeSharedInput>True</DesignTimeSharedInput>
</Compile>
</ItemGroup>
<ItemGroup>
<EmbeddedResource Include="My Project\Resources.resx">
<Generator>VbMyResourcesResXFileCodeGenerator</Generator>
<LastGenOutput>Resources.Designer.vb</LastGenOutput>
<CustomToolNamespace>My.Resources</CustomToolNamespace>
<SubType>Designer</SubType>
</EmbeddedResource>
</ItemGroup>
<ItemGroup>
<None Include=".editorconfig" />
<None Include="My Project\app.manifest" />
<None Include="My Project\Application.myapp">
<Generator>MyApplicationCodeGenerator</Generator>
<LastGenOutput>Application.Designer.vb</LastGenOutput>
</None>
<None Include="My Project\Settings.settings">
<Generator>SettingsSingleFileGenerator</Generator>
<CustomToolNamespace>My</CustomToolNamespace>
<LastGenOutput>Settings.Designer.vb</LastGenOutput>
</None>
<None Include="App.config" />
</ItemGroup>
<ItemGroup>
<Content Include="Content\Icons\RainbowIcon_48.ico" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\MyUtilities\PersonalUtilities\PersonalUtilities.vbproj">
<Project>{8405896b-2685-4916-bc93-1fb514c323a9}</Project>
<Name>PersonalUtilities</Name>
</ProjectReference>
<ProjectReference Include="..\SCrawler.Shared\SCrawler.Shared.vbproj">
<Project>{dc634700-24c7-42dd-bf8f-87e6cc54e625}</Project>
<Name>SCrawler.Shared</Name>
</ProjectReference>
</ItemGroup>
<Import Project="$(MSBuildToolsPath)\Microsoft.VisualBasic.targets" />
</Project>

View File

@@ -1,3 +1,3 @@
[*.vb]
# Modifier preferences
file_header_template = Copyright (C) 2023 Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>
file_header_template = Copyright (C) Andy https://github.com/AAndyProgram\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program. If not, see <https://www.gnu.org/licenses/>

View File

@@ -6,6 +6,10 @@
'
' This program is distributed in the hope that it will be useful,
' but WITHOUT ANY WARRANTY
Imports System.Drawing.Design
Imports System.ComponentModel
Imports PersonalUtilities.Tools.Grid.Attributes
Imports PersonalUtilities.Tools.Grid.EnumObjects
Namespace API.YouTube.Base
Public Structure Thumbnail : Implements IIndexable, IComparable(Of Thumbnail)
Public ID As String
@@ -24,8 +28,19 @@ Namespace API.YouTube.Base
End Structure
Public Structure Subtitles : Implements IIndexable, IComparable(Of Subtitles)
Public ID As String
Public Name As String
Private _Name As String
Public Property Name As String
Get
Dim n$ = _Name.IfNullOrEmpty(ID)
If CC Then n &= " (CC)"
Return n
End Get
Set(ByVal NewName As String)
_Name = NewName
End Set
End Property
Public Formats As String
Public CC As Boolean
Public ReadOnly Property FullID As String
Get
Return IIf(ID = "en", "en.*", ID)
@@ -47,9 +62,37 @@ Namespace API.YouTube.Base
Channel = 2
PlayList = 3
End Enum
<Editor(GetType(EnumDropDownEditor), GetType(UITypeEditor)), Flags>
Public Enum YouTubeChannelTab As Integer
<EnumValue(IsNullValue:=True)>
All = 0
Videos = 1
Shorts = 3
Playlists = 10
End Enum
<Editor(GetType(EnumDropDownEditor), GetType(UITypeEditor))>
Public Enum Protocols As Integer
<EnumValue(ExcludeFromList:=True)>
Undefined = -1
Any = 0
https = 1
m3u8 = 2
End Enum
<Editor(GetType(EnumDropDownEditor), GetType(UITypeEditor))>
Public Enum FileDateMode As Integer
None = 0
Before = 1
After = 2
End Enum
Public Enum M3U8CreationMode As Integer
Relative = 0
Absolute = 1
Both = 2
End Enum
Public Structure MediaObject : Implements IIndexable, IComparable(Of MediaObject)
Public Type As Plugin.UserMediaTypes
Public ID As String
Public ID_DRC As Boolean
Public Extension As String
Public Width As Integer
Public Height As Integer
@@ -58,7 +101,18 @@ Namespace API.YouTube.Base
''' <summary>Kb</summary>
Public Size As Double
Public Codec As String
Public Info As String
Public Protocol As String
Public ReadOnly Property ProtocolType As Protocols
Get
If Not Protocol.IsEmptyString Then
Select Case Protocol.StringToLower.StringTrim
Case "http", "https" : Return Protocols.https
Case "m3u8" : Return Protocols.m3u8
End Select
End If
Return Protocols.Undefined
End Get
End Property
Public URL As String
Public Property Index As Integer Implements IIndexable.Index
Private Function SetIndex(ByVal Obj As Object, ByVal Index As Integer) As Object Implements IIndexable.SetIndex
@@ -68,10 +122,14 @@ Namespace API.YouTube.Base
End Function
Private Function CompareTo(ByVal Other As MediaObject) As Integer Implements IComparable(Of MediaObject).CompareTo
If Type = Other.Type Then
If Width.CompareTo(Other.Width) = 0 Then
Return Size.CompareTo(Other.Size) * -1
If ID_DRC.CompareTo(Other.ID_DRC) = 0 Then
If Width.CompareTo(Other.Width) = 0 Then
Return Size.CompareTo(Other.Size) * -1
Else
Return Width.CompareTo(Other.Width) * -1
End If
Else
Return Width.CompareTo(Other.Width) * -1
Return ID_DRC.CompareTo(Other.ID_DRC)
End If
Else
Return CInt(Type).CompareTo(CInt(Other.Type))

View File

@@ -18,26 +18,93 @@ Namespace API.YouTube.Base
Public Const TrueUrlPattern As String = "https?://[^/]*?youtube.com/[^\?/&]+((\??[^\?/&]+|/[^\?/&]+))"
'2 - type; 5 - id
Public Const UrlTypePattern As String = "(?<=https?://[^/]*?youtube.com/)((@|[^\?/&]+))([/\?]{0,1}(list=|v=|)([^\?/&]*))(?=(\S+|\Z|))"
Public Const UrlTypePattern_BE As String = "(?<=https?://[^/]*?youtu.be/)([^\?/&]+)"
Private Sub New()
End Sub
Public Shared Function StandardizeURL(ByVal URL As String) As String
Try
URL = URL.StringTrim
Dim isMusic As Boolean = False, isShorts As Boolean = False
If Info_GetUrlType(URL, isMusic, isShorts) = YouTubeMediaType.Single Then
If Not isMusic And Not isShorts Then
Dim videoOptionRegex As RParams = RParams.DMS("[\?&]v=([^\?&]+)", 1, EDP.ReturnValue)
Dim data As List(Of String) = RegexReplace(URL, RParams.DMS(UrlTypePattern, 0, RegexReturn.ListByMatch, EDP.ReturnValue))
Dim val$ = String.Empty
If data.ListExists Then
For Each d$ In data
val = RegexReplace(d, videoOptionRegex)
If Not val.IsEmptyString Then Exit For
Next
data.Clear()
End If
If val.IsEmptyString Then val = RegexReplace(URL, RParams.DMS(UrlTypePattern_BE, 0, EDP.ReturnValue))
If Not val.IsEmptyString Then Return $"https://www.youtube.com/watch?v={val}"
End If
End If
Return URL
Catch ex As Exception
Return URL
End Try
End Function
Public Shared Function StandardizeURL_Channel(ByVal URL As String, Optional ByVal Process As Boolean = True) As String
Try
URL = URL.StringTrim
Dim ct As YouTubeChannelTab = YouTubeChannelTab.All
Dim isMusic As Boolean = False
If Process AndAlso Info_GetUrlType(URL, isMusic,,,, ct) = YouTubeMediaType.Channel AndAlso Not isMusic Then
If Not ct = YouTubeChannelTab.All Then
Dim rValue$ = String.Empty
Select Case ct
Case YouTubeChannelTab.Videos : rValue = "/videos"
Case YouTubeChannelTab.Shorts : rValue = "/shorts"
Case YouTubeChannelTab.Playlists : rValue = "/playlists"
End Select
If Not rValue.IsEmptyString Then
Dim startIndx% = InStr(URL, rValue)
If startIndx > 0 Then URL = URL.Remove(startIndx - 1)
End If
End If
End If
Return URL
Catch ex As Exception
Return URL
End Try
End Function
Public Shared Function IsMyUrl(ByVal URL As String) As Boolean
Return Not Info_GetUrlType(URL) = YouTubeMediaType.Undefined
End Function
Public Shared Function Info_GetUrlType(ByVal URL As String, Optional ByRef IsMusic As Boolean = False,
Optional ByRef IsChannelUser As Boolean = False, Optional ByRef Id As String = Nothing) As YouTubeMediaType
Public Shared Function Info_GetUrlType(ByVal URL As String, Optional ByRef IsMusic As Boolean = False, Optional ByRef IsShorts As Boolean = False,
Optional ByRef IsChannelUser As Boolean = False, Optional ByRef Id As String = Nothing,
Optional ByRef ChannelOptions As YouTubeChannelTab = YouTubeChannelTab.All) As YouTubeMediaType
URL = URL.StringTrim
If Not URL.IsEmptyString Then
IsMusic = URL.Contains("music.youtube.com")
IsChannelUser = False
IsShorts = False
Dim data As List(Of String) = RegexReplace(URL, RParams.DMS(UrlTypePattern, 0, RegexReturn.ListByMatch, EDP.ReturnValue))
If data.ListExists Then
If data.Count >= 6 Then Id = data(5)
If data.Count >= 3 And Not data(2).IsEmptyString Then
Select Case data(2).ToLower
Case "watch" : Return YouTubeMediaType.Single
Case "shorts" : IsShorts = True : Return YouTubeMediaType.Single
Case "playlist" : Return YouTubeMediaType.PlayList
Case UserChannelOption, "@" : IsChannelUser = data(2).ToLower = UserChannelOption : Return YouTubeMediaType.Channel
Case UserChannelOption, "@"
IsChannelUser = data(2).ToLower = UserChannelOption
If data.Count > 6 Then
Select Case data(6).StringToLower.StringTrimStart("/")
Case "videos" : ChannelOptions = YouTubeChannelTab.Videos
Case "shorts" : ChannelOptions = YouTubeChannelTab.Shorts
Case "playlists" : ChannelOptions = YouTubeChannelTab.Playlists
Case Else : ChannelOptions = YouTubeChannelTab.All
End Select
End If
Return YouTubeMediaType.Channel
End Select
End If
Else
Dim v$ = RegexReplace(URL, RParams.DMS(UrlTypePattern_BE, 0, EDP.ReturnValue))
If Not v.IsEmptyString Then Return YouTubeMediaType.Single
End If
End If
Return YouTubeMediaType.Undefined
@@ -57,27 +124,26 @@ Namespace API.YouTube.Base
''' <exception cref="InvalidOperationException"></exception>
Public Shared Function Parse(ByVal URL As String, Optional ByVal UseCookies As Boolean? = Nothing,
Optional ByVal Token As Threading.CancellationToken = Nothing, Optional ByVal Progress As IMyProgress = Nothing,
Optional ByVal GetDefault As Boolean? = Nothing, Optional ByVal GetShorts As Boolean? = Nothing,
Optional ByVal DateAfter As Date? = Nothing, Optional ByVal DateBefore As Date? = Nothing) As IYouTubeMediaContainer
Optional ByVal DateAfter As Date? = Nothing, Optional ByVal DateBefore As Date? = Nothing,
Optional ByVal ChannelOption As YouTubeChannelTab? = Nothing, Optional ByVal UrlAsIs As Boolean = False) As IYouTubeMediaContainer
URL = URL.StringTrim
If URL.IsEmptyString Then Throw New ArgumentNullException("URL", "URL cannot be null")
If Not MyYouTubeSettings.YTDLP.Value.Exists Then Throw New IO.FileNotFoundException("Path to 'yt-dlp.exe' not set or program not found at destination", MyYouTubeSettings.YTDLP.Value.ToString)
Dim urlOrig$ = URL
URL = RegexReplace(URL, TrueUrlRegEx)
If URL.IsEmptyString Then Throw New ArgumentNullException("URL", $"Can't get true URL from [{urlOrig}]")
Dim isMusic As Boolean = False
Dim objType As YouTubeMediaType = Info_GetUrlType(URL, isMusic)
Dim isMusic As Boolean = False, isShorts As Boolean = False
Dim channelTab As YouTubeChannelTab = YouTubeChannelTab.All
Dim objType As YouTubeMediaType = Info_GetUrlType(URL, isMusic, isShorts,,, channelTab)
If ChannelOption.HasValue Then channelTab = ChannelOption.Value
If Not objType = YouTubeMediaType.Undefined Then
Dim __GetDefault As Boolean = If(GetDefault, True)
Dim __GetShorts As Boolean = If(GetShorts, True)
If isMusic Then __GetShorts = False
Dim container As IYouTubeMediaContainer
Dim pattern$ = "%(channel_id)s_%(id)s_%(playlist_index)s"
Select Case objType
Case YouTubeMediaType.Single
__GetShorts = False
If isMusic Then container = New Track Else container = New Video
Case YouTubeMediaType.PlayList : container = New PlayList : pattern = "%(playlist_index)s_%(id)s" : __GetShorts = False
Case YouTubeMediaType.PlayList : container = New PlayList : pattern = "%(playlist_index)s_%(id)s"
Case YouTubeMediaType.Channel
container = New Channel
If isMusic Then pattern = "%(playlist_id)s/%(channel_id)s_%(id)s_%(playlist_index)s"
@@ -96,16 +162,16 @@ Namespace API.YouTube.Base
Dim useCookiesForce As Boolean = UseCookies.HasValue AndAlso UseCookies.Value AndAlso cookiesExists
If UseCookies.HasValue AndAlso UseCookies.Value Then
withCookieRequested = True
result = Parse_Internal(URL, pattern, _CachePathDefault, True, YouTubeCookieNetscapeFile, DateAfter, DateBefore, __GetDefault, __GetShorts)
result = Parse_Internal(URL, pattern, _CachePathDefault, True, YouTubeCookieNetscapeFile, DateAfter, DateBefore, objType, channelTab, isMusic, UrlAsIs)
End If
If Not result And Not withCookieRequested Then
If Not UseCookies.HasValue OrElse Not UseCookies.Value Then result = Parse_Internal(URL, pattern, _CachePathDefault, False, YouTubeCookieNetscapeFile, DateAfter, DateBefore, __GetDefault, __GetShorts)
If Not result And Not UseCookies.HasValue And cookiesExists Then result = Parse_Internal(URL, pattern, _CachePathDefault, True, YouTubeCookieNetscapeFile, DateAfter, DateBefore, __GetDefault, __GetShorts)
If Not UseCookies.HasValue OrElse Not UseCookies.Value Then result = Parse_Internal(URL, pattern, _CachePathDefault, False, YouTubeCookieNetscapeFile, DateAfter, DateBefore, objType, channelTab, isMusic, UrlAsIs)
If Not result And Not UseCookies.HasValue And cookiesExists Then result = Parse_Internal(URL, pattern, _CachePathDefault, True, YouTubeCookieNetscapeFile, DateAfter, DateBefore, objType, channelTab, isMusic, UrlAsIs)
End If
If result Then
container.Parse(Nothing, _CachePathDefault, isMusic, Token, Progress)
If Not container.HasError Then container.URL = URL : Return container
If Not container.HasError Then container.URL = URL.ToMusicUrl(isMusic) : container.IsShorts = isShorts : Return container
End If
container.Dispose()
End If
@@ -114,21 +180,40 @@ Namespace API.YouTube.Base
Private Shared Function Parse_Internal(ByVal URL As String, ByVal OutputPattern As String, ByVal OutputPath As SFile,
ByVal UseCookies As Boolean, ByVal CookiesFile As SFile,
ByVal DateAfter As Date?, ByVal DateBefore As Date?,
ByVal GetDefault As Boolean, ByVal GetShorts As Boolean) As Boolean
ByVal ObjType As YouTubeMediaType, ByVal ChannelTab As YouTubeChannelTab,
ByVal IsMusic As Boolean, ByVal UrlAsIs As Boolean) As Boolean
Try
Dim command$ = "yt-dlp --write-info-json --skip-download"
Dim command$ = $"{YTDLP_NAME} --write-info-json --skip-download"
command.StringAppend(GetCookiesCommand(UseCookies, CookiesFile), " ")
If DateAfter.HasValue Then command.StringAppend($"--dateafter {DateAfter.Value:yyyyMMdd}", " ")
If DateBefore.HasValue Then command.StringAppend($"--datebefore {DateBefore.Value:yyyyMMdd}", " ")
command.StringAppend("{0}" & $" -o ""{OutputPattern}""", " ")
'#If DEBUG Then
'Debug.WriteLine(String.Format(command, URL))
'#End If
Dim debugString As Func(Of String, String) = Function(ByVal input As String) As String
#If DEBUG Then
Debug.WriteLine(String.Format(command, URL))
Debug.WriteLine(input)
#End If
Return input
End Function
Using batch As New BatchExecutor(True)
With batch
.CommandPermanent = BatchExecutor.GetDirectoryCommand(MyYouTubeSettings.YTDLP.Value)
If GetDefault Then .Execute(String.Format(command, URL))
If GetShorts Then .Execute(String.Format(command, $"{URL.StringTrimEnd("/")}/shorts"))
If ObjType = YouTubeMediaType.Channel And Not IsMusic And Not UrlAsIs Then
Dim ct As List(Of YouTubeChannelTab) = EnumExtract(Of YouTubeChannelTab)(ChannelTab,, True).ListIfNothing
If ct.Count = 0 Then
.Execute(debugString(String.Format(command, $"{URL.StringTrimEnd("/")}/videos")))
.Execute(debugString(String.Format(command, $"{URL.StringTrimEnd("/")}/shorts")))
.Execute(debugString(String.Format(command, $"{URL.StringTrimEnd("/")}/playlists")))
Else
If ct.Contains(YouTubeChannelTab.Videos) Then .Execute(debugString(String.Format(command, $"{URL.StringTrimEnd("/")}/videos")))
If ct.Contains(YouTubeChannelTab.Shorts) Then .Execute(debugString(String.Format(command, $"{URL.StringTrimEnd("/")}/shorts")))
If ct.Contains(YouTubeChannelTab.Playlists) Then .Execute(debugString(String.Format(command, $"{URL.StringTrimEnd("/")}/playlists")))
End If
Else
.Execute(debugString(String.Format(command, URL)))
End If
End With
End Using
Return SFile.GetFiles(OutputPath,, IO.SearchOption.AllDirectories, EDP.ReturnValue).Count > 0

Some files were not shown because too many files have changed in this diff Show More