From f044f1321ab4404fc8cfa6e024fb67c268f12955 Mon Sep 17 00:00:00 2001 From: Andy <88590076+AAndyProgram@users.noreply.github.com> Date: Mon, 19 Jun 2023 06:06:30 +0300 Subject: [PATCH] 2023.6.19.0 --- Home.md | 6 +++-- Settings.md | 66 +++++++++++++++++++++++++++-------------------------- 2 files changed, 38 insertions(+), 34 deletions(-) diff --git a/Home.md b/Home.md index 65cc1ee..d0cb358 100644 --- a/Home.md +++ b/Home.md @@ -129,10 +129,12 @@ More about channels [here](https://github.com/AAndyProgram/SCrawler/wiki/Channel **[SITES REQUIREMENTS](https://github.com/AAndyProgram/SCrawler/wiki/Settings#sites-requirements)** -Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the `Saved posts user` textbox. Click OK. Go to the main window. +Go to Settings - Settings - Reddit/Instagram/Twitter/... Enter your username in the `Saved posts user` textbox. Click OK. Go to the main window. *If this option is not displayed in the site settings, it doesn't need to be configured.* This button looks like a bookmark. When you click on this button, a form for downloading saved posts will open. Saved posts are stored in the Reddit/Instagram/Twitter data path in the `!Saved posts` folder. +You may see a message like `Host [Site] is not ready` or `Host [Site] is unavailable`. This means that your credentials have expired, or some of your credentials are missing, or a saved posts for a particular site cannot be downloaded at this time. + ![Saved posts window](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SavedPosts.png) ### Download @@ -257,7 +259,7 @@ If you want to add a folder as a user, you need to use the following pattern: `D - `Favorite` - mark user as `Favorite`. - `Download images/videos` - for the created user, only the selected media types will be downloaded. - `Ready for download` - download this user in the operation `Download all`. -- `Get user media only` - this is an option for twitter only. Download all posts posted by this author. +- `Get user media only` - download all posts posted by this author. - `Add by list` - add users' profiles by list (new line as separator (`Shift+Enter` to insert new line)). - If you add users by the URL, you must check the `Auto detect site` checkbox; - If you are adding users by username, you must select the site in the combo box. diff --git a/Settings.md b/Settings.md index f0e3f5a..9932ae1 100644 --- a/Settings.md +++ b/Settings.md @@ -118,16 +118,17 @@ If you see something like `Jobs ` in the progress bar, it means that SCr - [Reddit](#reddit-requirements) - [Twitter](#twitter-requirements) -- [Mastodon](#Mastodon-requirements) +- [OnlyFans](#onlyfans-requirements) +- [Mastodon](#mastodon-requirements) - [Instagram](#instagram-requirements) - [RedGifs](#reddit-requirements) -- [YouTube](#YouTube-requirements) -- [Pinterest](#Pinterest-requirements) +- [YouTube](#youtube-requirements) +- [Pinterest](#pinterest-requirements) - [TikTok](#tiktok-requirements) - [PornHub](#pornhub-requirements) - [XHamster](#xhamster-requirements) - [XVIDEOS](#xvideos-requirements) -- [ThisVid](#ThisVid-requirements) +- [ThisVid](#thisvid-requirements) - [LPSG](#lpsg-requirements) ## Site default fields @@ -214,22 +215,22 @@ The Reddit parser can parse data without cookies, but you can add it if you like ![Twitter user settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsTwitterUser.png) -### How to find Twitter tokens -1. Open Google Chrome, Edge or FireFox. -1. Go to https://twitter.com -1. Press three-dots-button - More tools - Developer tools (or just press `Ctrl+Shift+I`). -1. Go to the network tab -1. In the window where you have twitter open, click on first post you see (**CLICK ON THE POST DATE**) -1. Then look at the dev tools and in the first lines you will see what you need! In the list on the left, find a query that starts with `TweetDetail?`... and click on it -1. Scroll down to `Request Headers` then find the fields below: - - `authorization: Bearer .......` - this is the authorization token. You need to copy the text from starting with the word `Bearer` to the end of the line and paste it into the `Authorization` field on the Twitter settings form. - - `x-csrf-token: ...............` - this is csrf token. You need to copy all the text after the words `x-csrf-token:` and paste it into the `Token` field on the Twitter settings form +## OnlyFans +- `user-id`, `x-bc`, `app-token`, `sec-sc-ua`, `UserAgent` - required headers ([how to find](#how-to-find-headers) (the request you need must start with `posts?limit=.....`)) +- `Use old authorization rules` - use old dynamic rules (from 'DATAHOARDERS') or new ones (from 'DIGITALCRIMINALS'). **Change this value only if you know what you are doing.** +- `Dynamic rules update` - 'Dynamic rules' update interval (minutes). Default: `1440`. +- `Dynamic rules` - overwrite 'Dynamic rules' with this URL. **Change this value only if you know what you are doing.** +![OnlyFans settings](https://raw.githubusercontent.com/AAndyProgram/SCrawler/main/ProgramScreenshots/SettingsSiteOnlyFans.png) + +### OnlyFans requirements +- [`Cookies`](#how-to-set-up-cookies) +- Headers ## Mastodon - `My Domain` - your account domain without `https://` (for example, `mastodon.social`) -- [`Authorization`](#how-to-find-twitter-tokens) - Bearer token. Must start with `Bearer ` word followed by a space. -- [`Token`](#how-to-find-twitter-tokens) - in this field you need to put `x-csrf-token`. +- `Authorization` - `Authorization` request header. Must start with `Bearer ` word. [How to find](#how-to-find-headers). +- `Token` - `x-csrf-token` request header. [How to find](#how-to-find-headers). - `Download GIFs` - *(default for new users) this can also be configured for a specific user.* - `GIFs special folder` - *(default for new users)* Put the GIFs in a special folder. **This is a folder name, not an absolute path** (examples: `SomeFolderName`, `SomeFolderName\SomeFolderName2`). This folder(s) will be created relative to the user's root folder. *This can also be configured for a specific user.* - `GIF prefix` - *(default for new users)* This prefix will be added to the beginning of the filename. *This can also be configured for a specific user.* @@ -240,9 +241,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like ### Mastodon requirements - `My Domain` -- [`Authorization`](#how-to-find-twitter-tokens) -- [`Token`](#how-to-find-twitter-tokens) -- [`Authorization`](#how-to-find-twitter-tokens) and [`Token`](#how-to-find-twitter-tokens) for each domain you want to download from (see [additional settings](#Mastodon-additional-setting)) +- [`Authorization`](#how-to-find-headers) +- [`Token`](#how-to-find-headers) +- [`Authorization`](#how-to-find-headers) and [`Token`](#how-to-find-headers) for each domain you want to download from (see [additional settings](#Mastodon-additional-setting)) ### Mastodon additional setting - `Domains` - select or add a new domain for which you want to add authorization @@ -254,8 +255,8 @@ The Reddit parser can parse data without cookies, but you can add it if you like ## Instagram - Authorization - `Hash` - in this field you need to put the hash of the Instagram session to download tagged posts ([how to find](#how-to-find-instagram-hash)). - - `x-csrftoken` - [how to find](#how-to-find-instagram-authorization-headers) - - `x-ig-app-id`, `ix-ig-www-claim`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-instagram-authorization-headers) + - `x-csrftoken` - [how to find](#how-to-find-headers) + - `x-ig-app-id`, `ix-ig-www-claim`, `sec-ch-ua`, `sec-ch-ua-full-version-list`, `sec-ch-ua-platform-version`, `UserAgent` - [how to find](#how-to-find-headers) - `Saved posts user` - your personal Instagram username to download your saved posts - Other parameters - `Download timeline` - Download timeline *(with this setting, you can simply enable/disable the downloading of some Instagram blocks)* @@ -274,9 +275,9 @@ The Reddit parser can parse data without cookies, but you can add it if you like ### Instagram requirements - **[COOKIES](#how-to-set-up-cookies) ARE REQUIRED ANYWAY** - [`Hash`](#how-to-find-instagram-hash) *(only for downloading tagged posts)* -- [`x-csrftoken`](#how-to-find-instagram-authorization-headers) -- [`x-ig-app-id`](#how-to-find-instagram-authorization-headers) -- [`ix-ig-www-claim`](#how-to-find-instagram-authorization-headers) +- [`x-csrftoken`](#how-to-find-headers) +- [`x-ig-app-id`](#how-to-find-headers) +- [`ix-ig-www-claim`](#how-to-find-headers) To download tagged posts you also need [`Hash`](#how-to-find-instagram-hash) @@ -295,14 +296,6 @@ To download tagged posts you also need [`Hash`](#how-to-find-instagram-hash) **NEVER use wrong credentials! I specially wrote there that there are two different credentials. Using the wrong credentials may result in your account being banned!** -### How to find Instagram authorization headers -1. Open browser developer tools -1. Go to `Network` -1. If this tab contain any data, clear it. -1. Open profile tagged photos. -1. Click `Ctrl+F` and find the text `x-ig-app-id` and `x-ig-www-claim`. -1. Copy the values and paste them into the corresponding fields on the Instagram settings form. - ### Instagram limits Instagram API is requests limited. For one request, the program receive only 50 posts. Before catching error 429, the program can process 200 requests. I reduced this to 195 requests and set a timer to wait for the next request after. This was added to bypass error 429 and prevent account ban. @@ -589,6 +582,15 @@ This command can be a batch command or any script file (bat, ps1 or whatever you 1. Paste the copied cookies text into the window that opens and click `OK`. 1. Click OK to close the cookies editor and save the cookies. +# How to find headers +1. Open Google Chrome, Microsoft Edge or FireFox. +1. Press three-dots-button - More tools - Developer tools (or just press `Ctrl+Shift+I`). +1. In the opened window, go to `Network`. +1. Go to the needed site and find a needed request. +1. Click on request. +1. Scroll down to `Request Headers`. +1. Find the headers you need, copy and paste them into the corresponding fields. + # How to find Imgur client ID 1. Open browser developer tools 1. Go to `Network`