2023.6.19.0

Home.md

@@ -129,10 +129,12 @@ More about channels [here](https://github.com/AAndyProgram/SCrawler/wiki/Channel
 
 **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)**
 
-Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the `Saved posts user` textbox. Click OK. Go to the main window.
+Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the `Saved posts user` textbox. Click OK. Go to the main window. *If this option is not displayed in the site settings, it doesn't need to be configured.*
 
 This button looks like a bookmark. When you click on this button, a form for downloading saved posts will open. Saved posts are stored in the Reddit/Instagram/Twitter data path in the `!Saved posts` folder.
 
+You may see a message like `Host [Site] is not ready` or `Host [Site] is unavailable`. This means that your credentials have expired, or some of your credentials are missing, or a saved posts for a particular site cannot be downloaded at this time.
+
 ![Saved posts window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SavedPosts.png)
 
 ### Download
@@ -257,7 +259,7 @@ If you want to add a folder as a user, you need to use the following pattern: `D
 - `Favorite` - mark user as `Favorite`.
 - `Download images/videos` - for the created user, only the selected media types will be downloaded.
 - `Ready for download` - download this user in the operation `Download all`.
-- `Get user media only` - this is an option for twitter only. Download all posts posted by this author.
+- `Get user media only` - download all posts posted by this author.
 - `Add by list` - add users' profiles by list (new line as separator (`Shift+Enter` to insert new line)).
   - If you add users by the URL, you must check the `Auto detect site` checkbox;
   - If you are adding users by username, you must select the site in the combo box.

Settings.md

@@ -118,16 +118,17 @@ If you see something like `Jobs <number>` in the progress bar, it means that SCr
 
 - [Reddit](#reddit-requirements)
 - [Twitter](#twitter-requirements)
-- [Mastodon](#Mastodon-requirements)
+- [OnlyFans](#onlyfans-requirements)
+- [Mastodon](#mastodon-requirements)
 - [Instagram](#instagram-requirements)
 - [RedGifs](#reddit-requirements)
-- [YouTube](#YouTube-requirements)
+- [YouTube](#youtube-requirements)
-- [Pinterest](#Pinterest-requirements)
+- [Pinterest](#pinterest-requirements)
 - [TikTok](#tiktok-requirements)
 - [PornHub](#pornhub-requirements)
 - [XHamster](#xhamster-requirements)
 - [XVIDEOS](#xvideos-requirements)
-- [ThisVid](#ThisVid-requirements)
+- [ThisVid](#thisvid-requirements)
 - [LPSG](#lpsg-requirements)
 
 ## Site default fields
@@ -214,22 +215,22 @@ The Reddit parser can parse data without cookies, but you can add it if you like
 
 ![Twitter user settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsTwitterUser.png)
 
-### How to find Twitter tokens
+## OnlyFans
-1. Open Google Chrome, Edge or FireFox.
+- `user-id`, `x-bc`, `app-token`, `sec-sc-ua`, `UserAgent` - required headers ([how to find](#how-to-find-headers) (the request you need must start with `posts?limit=.....`))
-1. Go to https://twitter.com
+- `Use old authorization rules` - use old dynamic rules (from 'DATAHOARDERS') or new ones (from 'DIGITALCRIMINALS'). **Change this value only if you know what you are doing.**
-1. Press three-dots-button - More tools - Developer tools (or just press `Ctrl+Shift+I`).
+- `Dynamic rules update` - 'Dynamic rules' update interval (minutes). Default: `1440`.
-1. Go to the network tab
+- `Dynamic rules` - overwrite 'Dynamic rules' with this URL. **Change this value only if you know what you are doing.**
-1. In the window where you have twitter open, click on first post you see (**CLICK ON THE POST DATE**)
-1. Then look at the dev tools and in the first lines you will see what you need! In the list on the left, find a query that starts with `TweetDetail?`... and click on it
-1. Scroll down to `Request Headers` then find the fields below:
-    - `authorization: Bearer .......` - this is the authorization token. You need to copy the text from starting with the word `Bearer` to the end of the line and paste it into the `Authorization` field on the Twitter settings form.
-    - `x-csrf-token: ...............` - this is csrf token. You need to copy all the text after the words `x-csrf-token:` and paste it into the `Token` field on the Twitter settings form
 
+![OnlyFans settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteOnlyFans.png)
+
+### OnlyFans requirements
+- [`Cookies`](#how-to-set-up-cookies)
+- Headers
 
 ## Mastodon
 - `My Domain` - your account domain without `https://` (for example, `mastodon.social`)
-- [`Authorization`](#how-to-find-twitter-tokens) - Bearer token. Must start with `Bearer ` word followed by a space.
+- `Authorization` - `Authorization` request header. Must start with `Bearer ` word. [How to find](#how-to-find-headers).
-- [`Token`](#how-to-find-twitter-tokens) - in this field you need to put `x-csrf-token`.
+- `Token` - `x-csrf-token` request header. [How to find](#how-to-find-headers).
 - `Download GIFs` - *(default for new users) this can also be configured for a specific user.*
 - `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.*
 - `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.*
@@ -240,9 +241,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like
 
 ### Mastodon requirements
 - `My Domain`
-- [`Authorization`](#how-to-find-twitter-tokens)
+- [`Authorization`](#how-to-find-headers)
-- [`Token`](#how-to-find-twitter-tokens)
+- [`Token`](#how-to-find-headers)
-- [`Authorization`](#how-to-find-twitter-tokens) and [`Token`](#how-to-find-twitter-tokens) for each domain you want to download from (see [additional settings](#Mastodon-additional-setting))
+- [`Authorization`](#how-to-find-headers) and [`Token`](#how-to-find-headers) for each domain you want to download from (see [additional settings](#Mastodon-additional-setting))
 
 ### Mastodon additional setting
 - `Domains` - select or add a new domain for which you want to add authorization
@@ -254,8 +255,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like
 ## Instagram
 - Authorization
   - `Hash` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how-to-find-instagram-hash)).
-  - `x-csrftoken` - [how to find](#how-to-find-instagram-authorization-headers)
+  - `x-csrftoken` - [how to find](#how-to-find-headers)
-  - `x-ig-app-id`, `ix-ig-www-claim`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-instagram-authorization-headers)
+  - `x-ig-app-id`, `ix-ig-www-claim`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-headers)
   - `Saved posts user` - your personal Instagram username to download your saved posts
 - Other parameters
   - `Download timeline` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)*
@@ -274,9 +275,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like
 ### Instagram requirements
 - **[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY**
 - [`Hash`](#how-to-find-instagram-hash) *(only for downloading tagged posts)*
-- [`x-csrftoken`](#how-to-find-instagram-authorization-headers)
+- [`x-csrftoken`](#how-to-find-headers)
-- [`x-ig-app-id`](#how-to-find-instagram-authorization-headers)
+- [`x-ig-app-id`](#how-to-find-headers)
-- [`ix-ig-www-claim`](#how-to-find-instagram-authorization-headers)
+- [`ix-ig-www-claim`](#how-to-find-headers)
 
 To download tagged posts you also need [`Hash`](#how-to-find-instagram-hash)
 
@@ -295,14 +296,6 @@ To download tagged posts you also need [`Hash`](#how-to-find-instagram-hash)
 
 **NEVER use wrong credentials! I specially wrote there that there are two different credentials. Using the wrong credentials may result in your account being banned!**
 
-### How to find Instagram authorization headers
-1. Open browser developer tools
-1. Go to `Network`
-1. If this tab contain any data, clear it.
-1. Open profile tagged photos.
-1. Click `Ctrl+F` and find the text `x-ig-app-id` and `x-ig-www-claim`.
-1. Copy the values and paste them into the corresponding fields on the Instagram settings form.
-
 ### Instagram limits
 
 Instagram API is requests limited. For one request, the program receive only 50 posts. Before catching error 429, the program can process 200 requests. I reduced this to 195 requests and set a timer to wait for the next request after. This was added to bypass error 429 and prevent account ban.
@@ -589,6 +582,15 @@ This command can be a batch command or any script file (bat, ps1 or whatever you
 1. Paste the copied cookies text into the window that opens and click `OK`.
 1. Click OK to close the cookies editor and save the cookies.
 
+# How to find headers
+1. Open Google Chrome, Microsoft Edge or FireFox.
+1. Press three-dots-button - More tools - Developer tools (or just press `Ctrl+Shift+I`).
+1. In the opened window, go to `Network`.
+1. Go to the needed site and find a needed request.
+1. Click on request.
+1. Scroll down to `Request Headers`.
+1. Find the headers you need, copy and paste them into the corresponding fields.
+
 # How to find Imgur client ID
 1. Open browser developer tools
 1. Go to `Network`