geonu/SCrawler
1
0
Fork 0
mirror of https://github.com/AAndyProgram/SCrawler.git synced 2026-03-14 15:52:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

2023.6.19.0

Andy
2023-06-19 06:06:30 +03:00
parent 0d7db54e05
commit f044f1321a
2 changed files with 38 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
Home.md

@@ -129,10 +129,12 @@ More about channels [here](https://github.com/AAndyProgram/SCrawler/wiki/Channel
**[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the `Saved posts user` textbox. Click OK. Go to the main window.
Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the `Saved posts user` textbox. Click OK. Go to the main window. *If this option is not displayed in the site settings, it doesn't need to be configured.*
This button looks like a bookmark. When you click on this button, a form for downloading saved posts will open. Saved posts are stored in the Reddit/Instagram/Twitter data path in the `!Saved posts` folder.
You may see a message like `Host [Site] is not ready` or `Host [Site] is unavailable`. This means that your credentials have expired, or some of your credentials are missing, or a saved posts for a particular site cannot be downloaded at this time.
![Saved posts window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SavedPosts.png)
### Download
@@ -257,7 +259,7 @@ If you want to add a folder as a user, you need to use the following pattern: `D
- `Favorite` - mark user as `Favorite`.
- `Download images/videos` - for the created user, only the selected media types will be downloaded.
- `Ready for download` - download this user in the operation `Download all`.
- `Get user media only` - this is an option for twitter only. Download all posts posted by this author.
- `Get user media only` - download all posts posted by this author.
- `Add by list` - add users' profiles by list (new line as separator (`Shift+Enter` to insert new line)).
- If you add users by the URL, you must check the `Auto detect site` checkbox;
- If you are adding users by username, you must select the site in the combo box.

66
Settings.md

@@ -118,16 +118,17 @@ If you see something like `Jobs <number>` in the progress bar, it means that SCr
- [Reddit](#reddit-requirements)
- [Twitter](#twitter-requirements)
- [Mastodon](#Mastodon-requirements)
- [OnlyFans](#onlyfans-requirements)
- [Mastodon](#mastodon-requirements)
- [Instagram](#instagram-requirements)
- [RedGifs](#reddit-requirements)
- [YouTube](#YouTube-requirements)
- [Pinterest](#Pinterest-requirements)
- [YouTube](#youtube-requirements)
- [Pinterest](#pinterest-requirements)
- [TikTok](#tiktok-requirements)
- [PornHub](#pornhub-requirements)
- [XHamster](#xhamster-requirements)
- [XVIDEOS](#xvideos-requirements)
- [ThisVid](#ThisVid-requirements)
- [ThisVid](#thisvid-requirements)
- [LPSG](#lpsg-requirements)
## Site default fields
@@ -214,22 +215,22 @@ The Reddit parser can parse data without cookies, but you can add it if you like
![Twitter user settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsTwitterUser.png)
### How to find Twitter tokens
1. Open Google Chrome, Edge or FireFox.
1. Go to https://twitter.com
1. Press three-dots-button - More tools - Developer tools (or just press `Ctrl+Shift+I`).
1. Go to the network tab
1. In the window where you have twitter open, click on first post you see (**CLICK ON THE POST DATE**)
1. Then look at the dev tools and in the first lines you will see what you need! In the list on the left, find a query that starts with `TweetDetail?`... and click on it
1. Scroll down to `Request Headers` then find the fields below:
- `authorization: Bearer .......` - this is the authorization token. You need to copy the text from starting with the word `Bearer` to the end of the line and paste it into the `Authorization` field on the Twitter settings form.
- `x-csrf-token: ...............` - this is csrf token. You need to copy all the text after the words `x-csrf-token:` and paste it into the `Token` field on the Twitter settings form
## OnlyFans
- `user-id`, `x-bc`, `app-token`, `sec-sc-ua`, `UserAgent` - required headers ([how to find](#how-to-find-headers) (the request you need must start with `posts?limit=.....`))
- `Use old authorization rules` - use old dynamic rules (from 'DATAHOARDERS') or new ones (from 'DIGITALCRIMINALS'). **Change this value only if you know what you are doing.**
- `Dynamic rules update` - 'Dynamic rules' update interval (minutes). Default: `1440`.
- `Dynamic rules` - overwrite 'Dynamic rules' with this URL. **Change this value only if you know what you are doing.**
![OnlyFans settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteOnlyFans.png)
### OnlyFans requirements
- [`Cookies`](#how-to-set-up-cookies)
- Headers
## Mastodon
- `My Domain` - your account domain without `https://` (for example, `mastodon.social`)
- [`Authorization`](#how-to-find-twitter-tokens) - Bearer token. Must start with `Bearer ` word followed by a space.
- [`Token`](#how-to-find-twitter-tokens) - in this field you need to put `x-csrf-token`.
- `Authorization` - `Authorization` request header. Must start with `Bearer ` word. [How to find](#how-to-find-headers).
- `Token` - `x-csrf-token` request header. [How to find](#how-to-find-headers).
- `Download GIFs` - *(default for new users) this can also be configured for a specific user.*
- `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.*
- `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.*
@@ -240,9 +241,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like
### Mastodon requirements
- `My Domain`
- [`Authorization`](#how-to-find-twitter-tokens)
- [`Token`](#how-to-find-twitter-tokens)
- [`Authorization`](#how-to-find-twitter-tokens) and [`Token`](#how-to-find-twitter-tokens) for each domain you want to download from (see [additional settings](#Mastodon-additional-setting))
- [`Authorization`](#how-to-find-headers)
- [`Token`](#how-to-find-headers)
- [`Authorization`](#how-to-find-headers) and [`Token`](#how-to-find-headers) for each domain you want to download from (see [additional settings](#Mastodon-additional-setting))
### Mastodon additional setting
- `Domains` - select or add a new domain for which you want to add authorization
@@ -254,8 +255,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
## Instagram
- Authorization
- `Hash` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how-to-find-instagram-hash)).
- `x-csrftoken` - [how to find](#how-to-find-instagram-authorization-headers)
- `x-ig-app-id`, `ix-ig-www-claim`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-instagram-authorization-headers)
- `x-csrftoken` - [how to find](#how-to-find-headers)
- `x-ig-app-id`, `ix-ig-www-claim`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-headers)
- `Saved posts user` - your personal Instagram username to download your saved posts
- Other parameters
- `Download timeline` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
@@ -274,9 +275,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like
### Instagram requirements
- **[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY**
- [`Hash`](#how-to-find-instagram-hash) *(only for downloading tagged posts)*
- [`x-csrftoken`](#how-to-find-instagram-authorization-headers)
- [`x-ig-app-id`](#how-to-find-instagram-authorization-headers)
- [`ix-ig-www-claim`](#how-to-find-instagram-authorization-headers)
- [`x-csrftoken`](#how-to-find-headers)
- [`x-ig-app-id`](#how-to-find-headers)
- [`ix-ig-www-claim`](#how-to-find-headers)
To download tagged posts you also need [`Hash`](#how-to-find-instagram-hash)
@@ -295,14 +296,6 @@ To download tagged posts you also need [`Hash`](#how-to-find-instagram-hash)
**NEVER use wrong credentials! I specially wrote there that there are two different credentials. Using the wrong credentials may result in your account being banned!**
### How to find Instagram authorization headers
1. Open browser developer tools
1. Go to `Network`
1. If this tab contain any data, clear it.
1. Open profile tagged photos.
1. Click `Ctrl+F` and find the text `x-ig-app-id` and `x-ig-www-claim`.
1. Copy the values and paste them into the corresponding fields on the Instagram settings form.
### Instagram limits
Instagram API is requests limited. For one request, the program receive only 50 posts. Before catching error 429, the program can process 200 requests. I reduced this to 195 requests and set a timer to wait for the next request after. This was added to bypass error 429 and prevent account ban.
@@ -589,6 +582,15 @@ This command can be a batch command or any script file (bat, ps1 or whatever you
1. Paste the copied cookies text into the window that opens and click `OK`.
1. Click OK to close the cookies editor and save the cookies.
# How to find headers
1. Open Google Chrome, Microsoft Edge or FireFox.
1. Press three-dots-button - More tools - Developer tools (or just press `Ctrl+Shift+I`).
1. In the opened window, go to `Network`.
1. Go to the needed site and find a needed request.
1. Click on request.
1. Scroll down to `Request Headers`.
1. Find the headers you need, copy and paste them into the corresponding fields.
# How to find Imgur client ID
1. Open browser developer tools
1. Go to `Network`