geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2025.1.12.0

Andy
2025-01-12 23:17:45 +03:00
parent 13782ab7aa
commit 4234985160
2 changed files with 52 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
Home.md

@@ -524,11 +524,7 @@ The program implements two ways to add users to this list:
# Move the SCrawler data to a new location. # Move the SCrawler data to a new location.
1. Close SCrawler. 1. Close SCrawler.
1. Move the data to a new location. 1. Move the data to a new location.
1. In the SCrawler settings folder, find the `Settings.xml` file. 1. In the SCrawler settings folder, find the `Settings.xml` file. Change the value of the `GlobalPath` node.
1. Change the value of the `GlobalPath` node. 1. Open each `Host_..._..._Name.xml` and `Responser_..._Settings.xml` file and update the `Path` and `SavedPostsPath` values to the new locations.
1. For each site in this file (under the `Sites` node) where it exists, change the value of the `Path` node to the new location. Example: `Sites` - `Instagram` - `Path`. 1. Open the `Users.xml` file (in the SCrawler settings folder). Check the `SpecialPath` and `SpecialCollectionPath` values to make sure they still point to the correct locations. If not, replace these values with the correct ones. To find all non-empty `SpecialPath` and `SpecialCollectionPath` values, you can use a simple regular expressions: `SpecialPath="[^"]+"`, `SpecialCollectionPath="[^"]+"`.
1. Save this file and close.
1. Open the `Users.xml` file (in the SCrawler settings folder).
1. Check the `SpecialPath` values to make sure they still point to the correct locations. If not, replace these values with the correct ones. To find all non-empty `SpecialPath` values, you can use a simple regular expression: `SpecialPath="[^"]+"`.
1. Save this file and close.
1. Open SCrawler. 1. Open SCrawler.

64
Settings.md

@@ -217,7 +217,7 @@ ffmpeg is required for several sites like Reddit, PornHub, XHamster, XVIDEOS, et
### Gallery-dl ### Gallery-dl
Version **1.27.7** ([release](https://github.com/mikf/gallery-dl/releases/tag/v1.27.7); [exe](https://github.com/mikf/gallery-dl/releases/download/v1.27.7/gallery-dl.exe)) Version **1.28.3** ([release](https://github.com/mikf/gallery-dl/releases/tag/v1.28.3); [exe](https://github.com/mikf/gallery-dl/releases/download/v1.28.3/gallery-dl.exe))
**The following sites use gallery-dl:** **The following sites use gallery-dl:**
- [Pinterest](#pinterest) - [Pinterest](#pinterest)
@@ -225,10 +225,10 @@ Version **1.27.7** ([release](https://github.com/mikf/gallery-dl/releases/tag/v1
### YT-DLP ### YT-DLP
Version **2024.11.18** ([release](https://github.com/yt-dlp/yt-dlp/releases/tag/2024.11.18)) Version **2024.12.23** ([release](https://github.com/yt-dlp/yt-dlp/releases/tag/2024.12.23))
- x64 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2024.11.18/yt-dlp.exe) - x64 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2024.12.23/yt-dlp.exe)
- x86 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2024.11.18/yt-dlp_x86.exe) - x86 version - [exe](https://github.com/yt-dlp/yt-dlp/releases/download/2024.12.23/yt-dlp_x86.exe)
**The following sites use yt-dlp:** **The following sites use yt-dlp:**
- [YouTube](#youtube) - [YouTube](#youtube)
@@ -237,10 +237,7 @@ Version **2024.11.18** ([release](https://github.com/yt-dlp/yt-dlp/releases/tag/
### OF-Scraper ### OF-Scraper
Version **3.10.7** ([release](https://github.com/datawhores/OF-Scraper/releases/tag/3.10.7)) Version **3.12.9** ([release](https://github.com/datawhores/OF-Scraper/releases/tag/3.12.9))
<!--
Version **2023.12.25_485586_main_d584b2b_package** ([release](https://github.com/datawhores/OF-Scraper/releases/tag/d584b2b))
-->
**Don't download the `exe` file. Always download `zip`.** **Don't download the `exe` file. Always download `zip`.**
@@ -365,6 +362,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
- `mp4decrypt path` - the path to the `mp4decrypt.exe` - `mp4decrypt path` - the path to the `mp4decrypt.exe`
- `key-mode-default` - change the key-mode. Default: `cdrm`. **Change this value only if you know what you are doing.** - `key-mode-default` - change the key-mode. Default: `cdrm`. **Change this value only if you know what you are doing.**
- `keydb_api` - change the keydb_api. **Change this value only if you know what you are doing.** - `keydb_api` - change the keydb_api. **Change this value only if you know what you are doing.**
- `Private key` - path to the DRM key file `private_key.pem`
- `Client ID` - path to the DRM key file `client_id.bin`
![OnlyFans settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteOnlyFans.png) ![OnlyFans settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteOnlyFans.png)
@@ -400,10 +399,14 @@ Other settings **(change these values only if you know what you are doing)**
1. Open OnlyFans site settings 1. Open OnlyFans site settings
1. Add the path to the **ofscraper.exe** 1. Add the path to the **ofscraper.exe**
1. Add the path to the **mp4decrypt.exe** 1. Add the path to the **mp4decrypt.exe**
1. Fill in the paths to your DRM keys (if you use them) ([how to get them](https://forum.videohelp.com/threads/408031-Dumping-Your-own-L3-CDM-with-Android-Studio))
1. If you use DRM keys, set `key-mode-default` option to `manual`
1. Click OK 1. Click OK
*[OF-Scraper guide](https://of-scraper.gitbook.io/of-scraper/)* *[OF-Scraper guide](https://of-scraper.gitbook.io/of-scraper/)*
**DRM keys**: [OF-Scraper reference](https://of-scraper.gitbook.io/of-scraper/cdm-options#manual), **[complete guide](https://forum.videohelp.com/threads/408031-Dumping-Your-own-L3-CDM-with-Android-Studio)**
### OnlyFans FAQ ### OnlyFans FAQ
> How to download videos? > How to download videos?
@@ -674,13 +677,18 @@ The `User ID` and `UserAgent` values must be obtained from the page `https://jus
--> -->
## YouTube ## YouTube
- `Download user videos` - Authorization
- `Download user shorts` - `Use cookies` - default value for new users. Use cookies to download the user.
- `Download user playlists` - New user defaults
- `Download user community: images` - `Download user videos`
- `Download user community: videos` - `Download user shorts`
- `Ignore community errors` - if true, community errors will not be added to the log. - `Download user playlists`
- `Use cookies` - default value for new users. Use cookies to download the user. - `Download user community: images`
- `Download user community: videos`
- Communities
- `YouTube API host` - YouTube API instance host (YouTube-operational-API). Example: `localhost/YouTube-operational-API`, `http://localhost/YouTube-operational-API` ([how to create](#youtube-operational-api))
- `YouTube API key` - YouTube Data API v3 developer key
- `Ignore community errors` - if true, community errors will not be added to the log.
See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader#Settings). See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/YouTube-downloader#Settings).
@@ -691,6 +699,32 @@ See additional settings [here](https://github.com/AAndyProgram/SCrawler/wiki/You
- [**yt-dlp**](#YT-DLP) - [**yt-dlp**](#YT-DLP)
- [**ffmpeg**](#ffmpeg) - [**ffmpeg**](#ffmpeg)
### YouTube operational API
To download communities, you need to install the `YouTube operational API`.
The complete guide is [here](https://github.com/Benjamin-Loison/YouTube-operational-API/blob/476ae90317150deb2e187eb2dd2b59afc5b384a6/README.md#install-your-own-instance-of-the-api).
1. Download and run [WampServer 3](https://sourceforge.net/projects/wampserver/files/latest/download). To install WampServer you'll need `VC Redistributable Packages`. You can download them all from [here](https://wampserver.aviatechno.net/) ([x86](https://wampserver.aviatechno.net/files/vcpackages/all_vc_redist_x86.zip), [x86 x64](https://wampserver.aviatechno.net/files/vcpackages/all_vc_redist_x86_x64.zip)). Install each package 2010, 2012, 2013, 2015.
2. Open command prompt
3. Type `cd C:\wamp64\www\`
4. Clone this repository by using `git clone https://github.com/Benjamin-Loison/YouTube-operational-API`. To use this command you'll need `Git`. You can download it from [here](https://git-scm.com/downloads/win).
5. Close command prompt
6. Download [composer](https://github.com/composer/windows-setup/releases/latest)
7. Download [protoc](https://github.com/protocolbuffers/protobuf/releases/latest)
8. Extract `protoc` files to `c:\wamp64\www\YouTube-operational-API\protoc\`
9. Add `c:\wamp64\www\YouTube-operational-API\protoc\bin\` to system paths
10. Install `composer` with default options
11. Open command prompt
12. Type `c:\wamp64\www\YouTube-operational-API\`
13. Enter the following command
```bash
for /f "usebackq tokens=*" %a in (`dir /S /B "c:/wamp64/www/YouTube-operational-API/protoc/include/google/protobuf/"`) do protoc --php_out=proto/php/ --proto_path="c:/wamp64/www/YouTube-operational-API/protoc/include/google/protobuf/" %a
```
14. Launch WampServer
15. Verify that your API instance is reachable by trying to access: http://localhost/YouTube-operational-API/
16. Add this URL to SCrawler YouTube settings `YouTube API host`
## Pinterest ## Pinterest
- `Concurrent downloads` - the number of concurrent downloads. - `Concurrent downloads` - the number of concurrent downloads.
- `Saved posts user` - personal profile username. - `Saved posts user` - personal profile username.