feat: proper handle api hits (#2)

.github/ISSUE_TEMPLATE/bug_report.md

@@ -18,7 +18,7 @@ A clear and concise description of what you expected to happen.
 
 **Desktop (please complete the following information):**
  - Python version
- - Use default jikan api or not
+ - Use default jikan url api or not
 
 **Additional context**
 Add any other context about the problem here.

.github/workflows/get_test.yml

@@ -1,4 +1,4 @@
-name: Test get requests
+name: Test get
 on:
   push:
     branches:

.github/workflows/search_test.yml

@@ -1,4 +1,4 @@
-name: Test search requests
+name: Test search
 on:
   push:
     branches:

README.md

@@ -3,16 +3,17 @@
 
 <h4 align="center">Python client for Jikan.moe, simplified with amplified in mind.</h4>
 <p align="center">
-	<a href="https://github.com/ScathachGrip/jikan4snek/actions/workflows/request_sequentially.yml"><img src="https://github.com/ScathachGrip/jikan4snek/workflows/Request%20Sequentially/badge.svg"></a>
+	<a href="https://github.com/ScathachGrip/jikan4snek/actions/workflows/get_test.yml"><img src="https://github.com/ScathachGrip/jikan4snek/workflows/Test%20get/badge.svg"></a>
+  	<a href="https://github.com/ScathachGrip/jikan4snek/actions/workflows/search_test.yml"><img src="https://github.com/ScathachGrip/jikan4snek/workflows/Test%20search/badge.svg"></a>
 	<a href="https://codeclimate.com/github/ScathachGrip/jikan4snek/maintainability"><img src="https://api.codeclimate.com/v1/badges/1318c78a4b9911edf844/maintainability" /></a>
 </p>
 
 
 The motivation is simplified the api call, customizable behaviour, and user should have no worries with ratelimit.  
-Jikan4snek simulating the requests with saved cache and apply coroutine delay if cache was expired.
+Jikan4snek simulating requests with cache and apply coroutine delay if ratelimit was hit or it's expired.
 
 <a href="https://github.com/ScathachGrip/jikan4snek/blob/master/CONTRIBUTING.md">Contributing</a> •
-<a href="https://github.com/ScathachGrip/jikan4snek/wiki/Routing">Documentation</a> •
+<a href="https://scathachgrip.github.io/jikan4snek">Documentation</a> •
 <a href="https://github.com/ScathachGrip/jikan4snek/issues/new/choose">Report Issues</a>
 </div>
 
@@ -27,6 +28,7 @@ Jikan4snek simulating the requests with saved cache and apply coroutine delay if
     - [Usage](#usage)
       - [Get](#get)
       - [Search](#search)
+      - [Bulk-request](#bulk-request)
     - [Constructors](#constructors)
     - [Running tests](#running-tests)
   - [Debug](#debug)
@@ -76,7 +78,7 @@ asyncio.run(main())
 ```
 
 ### Constructors
-You can apply your own instance of [Jikan](https://github.com/jikan-me/jikan-rest), user-agent, sqlite backend, and cache expiration, and debug on the constructor.
+You can apply your own instance of [Jikan](https://github.com/jikan-me/jikan-rest), user-agent, sqlite backend, cache expiration, and debug.
 
 The default:
 ```py
@@ -251,19 +253,67 @@ await jikan.search("sin", limit=10, page=1).users()
 ```
 </details>
 
+### Bulk request
+If you using `asyncio.gather(*[jikan.get("..").anime()])` sadly, It will broke the base api hit, Just do it like usual, enable debug, and snek will handle the ratelimit for you. For example:
+
+```py
+import asyncio
+import jikan4snek
+
+some_bunch_of_anime = [
+    44511, 50602, 50172, 49918, 49596, 41467, 
+    48316, 49709, 42962, 47917, 49891, 50425, 
+    50710, 49784, 51098, 49979, 52046, 52193, 
+    51128, 48542, 49828, 51403, 50205, 50528, 
+    50590, 51212, 30455, 50923, 50348, 51680]
+
+async def main():
+    Jikan = jikan4snek.Jikan4SNEK(debug=True)
+    for i in some_bunch_of_anime:
+        res = await Jikan.get(i).anime()
+        print("Anime:", res['data']['title'])
+
+asyncio.run(main())
+```
+
 ### Running tests
 Check workflows and the whole `/test` folder.
 
 ## Debug
-Enable debug ratelimit hit. Default is False. Use this if jikan4snek request is not working as expected.  
+Enable debug. Default is False. Use this if jikan4snek request is not working as expected.  
 ```py
 jikan = Jikan4SNEK(debug=True)
 ```
 
-<img src="https://cdn.discordapp.com/attachments/1046495201176334467/1054231070616342558/SNEKLOGGG_1.png" width="600" alt="sneklog">  
+```
+2022-12-20 21:02:06,435 | INFO | request:fetch | Not hitting API, cache is available
+2022-12-20 21:02:06,435 | INFO | request:fetch | is_cache:True | status_code:200 | url:https://api.jikan.moe/v4/anime/44511 | took:0.02 seconds
+2022-12-20 21:02:06,435 | INFO | main:memek | Anime: Chainsaw Man
+2022-12-20 21:02:06,435 | INFO | request:fetch | Not hitting API, cache is available
+2022-12-20 21:02:06,435 | INFO | request:fetch | is_cache:True | status_code:200 | url:https://api.jikan.moe/v4/anime/50602 | took:0.0 seconds
+2022-12-20 21:02:06,435 | INFO | main:memek | Anime: Spy x Family Part 2
+2022-12-20 21:02:06,435 | INFO | request:fetch | Not hitting API, cache is available
+2022-12-20 21:02:06,435 | INFO | request:fetch | is_cache:True | status_code:200 | url:https://api.jikan.moe/v4/anime/50172 | took:0.0 seconds
+2022-12-20 21:02:06,435 | INFO | main:memek | Anime: Mob Psycho 100 III
+2022-12-20 21:02:07,997 | INFO | request:fetch | First conditions of request, API hit 1
+2022-12-20 21:02:07,997 | INFO | request:fetch | is_cache:False | status_code:200 | url:https://api.jikan.moe/v4/anime/49918 | took:1.56 seconds
+2022-12-20 21:02:07,997 | INFO | main:memek | Anime: Boku no Hero Academia 6th Season
+2022-12-20 21:02:09,560 | INFO | request:fetch | Second conditions of request, API hit 2
+2022-12-20 21:02:09,576 | INFO | request:fetch | is_cache:False | status_code:200 | url:https://api.jikan.moe/v4/anime/49596 | took:0.016 seconds
+2022-12-20 21:02:09,576 | INFO | main:memek | Anime: Blue Lock
+2022-12-20 21:02:11,122 | INFO | request:fetch | Third conditions of request, apply sleep for 1 seconds..
+2022-12-20 21:02:12,123 | INFO | request:fetch | Third should back to first condtions, API hit 1
+2022-12-20 21:02:12,154 | INFO | request:fetch | is_cache:False | status_code:200 | url:https://api.jikan.moe/v4/anime/41467 | took:0.031 seconds
+2022-12-20 21:02:12,154 | INFO | main:memek | Anime: Bleach: Sennen Kessen-hen
+2022-12-20 21:02:13,716 | INFO | request:fetch | Second conditions of request, API hit 2
+2022-12-20 21:02:13,732 | INFO | request:fetch | is_cache:False | status_code:200 | url:https://api.jikan.moe/v4/anime/48316 | took:0.016 seconds
+2022-12-20 21:02:13,732 | INFO | main:memek | Anime: Kage no Jitsuryokusha ni Naritakute!
+2022-12-20 21:02:15,247 | INFO | request:fetch | Third conditions of request, apply sleep for 1 seconds..
+2022-12-20 21:02:16,247 | INFO | request:fetch | Third should back to first condtions, API hit 1
+```
 
 ### Jikan4snek.dump
-Short hand of [json.dump()](https://docs.python.org/3/library/json.html#json.dumps) If you are phobia with arbitrary bad indentation of json, use `Jikan4snek.dump()` to dump them, It's definitely str, not dictionaries, just in case for reading object to save your time.
+Short hand of [json.dump()](https://docs.python.org/3/library/json.html#json.dumps) If you are phobia with arbitrary bad indentation of json. Use `Jikan4snek.dump()` to dump them, It's definitely str, not dictionaries, just in case for reading object to save your time.
 
 ## Documentation
 https://scathachgrip.github.io/jikan4snek

jikan4snek/base/constant.py

@@ -1,8 +1,11 @@
 import json
 import re
-from jikan4snek import __version__
+import time
 import logging
-logging.basicConfig(level=logging.INFO, format='%(asctime)s | %(levelname)s | %(funcName)s:%(module)s | %(message)s')
+from jikan4snek import __version__
+from aiohttp_client_cache import CachedSession
+logging.basicConfig(level=logging.INFO,
+                    format='%(asctime)s | %(levelname)s | %(funcName)s:%(module)s | %(message)s')
 
 
 class Api:
@@ -12,7 +15,6 @@ def __init__(
         BASE_STRICT_DELAY=False,
         BASE_DEBUG=False,
         BASE_CONSTANT_HIT=2,
-        BASE_SIMULATE_HIT=1.3,
         BASE_EXPIRE_CACHE=60,
         BASE_SQLITE_BACKEND="jikan4snek_cache/jikan4snek",
         BASE_headers: dict = {
@@ -94,7 +96,6 @@ def __init__(
         self.strict_delay = BASE_STRICT_DELAY
         self.debug = BASE_DEBUG
         self.constant_hit = BASE_CONSTANT_HIT
-        self.simulate_hit = BASE_SIMULATE_HIT
         self.expire_cache = BASE_EXPIRE_CACHE
         self.sqlite_backend = BASE_SQLITE_BACKEND
         self.headers = BASE_headers
@@ -140,9 +141,54 @@ def resolve(b_object: dict) -> dict:
     """
     return json.loads(b_object)
 
+
 def rm_slash(url: str) -> str:
+    """Removes multiple slash from the url.
+
+    Parameters
+    ----------
+    url : str
+        Some url with multiple slash.
+
+    Returns
+    -------
+    str
+        The url with only one slash.
+    """
     return re.sub(r"(?<!:)/{2,}", "/", url)
 
-def logs(info: str) -> None:
-    return logging.info(info)
 
+async def fetch_hit(cache, endpoint, headers) -> dict:
+    """Fetches the data from the endpoint.
+
+    Parameters
+    ----------
+    cache : CacheBackend
+        SQLite cache backend.
+    endpoint : str
+        The jikan endpoint.
+    headers : dict
+        The headers for the request.
+
+    Returns
+    -------
+    dict
+        The data, status code, and url.
+    """
+    start_fetch = time.time()
+    async with CachedSession(cache=cache) as session_hit:
+        async with session_hit.get(endpoint, headers=headers) as resp_hit:
+            time_took = time.time() - start_fetch
+            time_took = round(time_took, 3)
+
+            data = await resp_hit.json()
+            status = resp_hit.status
+            url = resp_hit.url
+
+            return {
+                "reparse": data,
+                "status": status,
+                "url": url,
+                "cache": False,
+                "took": time_took
+            }

jikan4snek/base/fetch.py

@@ -1,9 +1,9 @@
 import time
 import logging
-from asyncio import sleep
+import asyncio
 from aiohttp_client_cache import CachedSession, SQLiteBackend
 from typing import Union
-from .constant import Api, rm_slash
+from .constant import Api, rm_slash, fetch_hit
 
 Jikan = Api()
 api_hit = []
@@ -54,7 +54,7 @@ async def request(
         urls_expire_after={"*/random": 0},
     )
 
-    ## if someid has "?q=" then it's a search
+    # if someid has "?q=" then it's a search
     if "?q=" in str(path):
         endpoint = f"{path}"
 
@@ -64,56 +64,62 @@ async def request(
     start_fetch = time.time()
     async with CachedSession(cache=sequel_cfg) as session:
         async with session.get(f"{api}/{rm_slash(endpoint)}", headers=ua) as resp:
-            simulate_time = time.time() - start_fetch
-            simulate_time = round(simulate_time, 2)
-
-            if debug:
-                logging.info(
-                    f"Cache:{resp.from_cache} | Status:{resp.status} | {resp.url}"
-                )
-                logging.info(
-                    f"Add delay hit depends on your internet, took {simulate_time} sec."
-                )
-
-            try:
-                if Jikan.strict_delay:
-                    if resp.from_cache:
-                        res = await resp.json()
-                        return res
-                    else:
-                        res = await resp.json()
-                        await sleep(Jikan.constant_hit)
-                        return res
+            time_took = time.time() - start_fetch
+            time_took = round(time_took, 3)
+            if resp.from_cache:
+                if debug:
+                    logging.info(f"Not hitting API, cache is available")
+                    logging.info(
+                        f"is_cache:{resp.from_cache} | status_code:{resp.status} | url:{resp.url} | took:{time_took} seconds")
+
+                return await resp.json()
+
+            else:
+                if len(api_hit) == 0:
+                    api_hit.append(1)
+                    if debug:
+                        logging.info(
+                            f"First conditions of request, API hit {len(api_hit)}")
+                        logging.info(
+                            f"is_cache:{resp.from_cache} | status_code:{resp.status} | url:{resp.url} | took:{time_took} seconds")
+
+                    return await resp.json()
+
+                elif len(api_hit) <= 1:
+                    api_hit.append(1)
+                    if debug:
+                        logging.info(
+                            f"Second conditions of request, API hit {len(api_hit)}")
+
+                    api = await fetch_hit(
+                        cache=sequel_cfg,
+                        endpoint=f"{api}/{rm_slash(endpoint)}",
+                        headers=ua,
+                    )
+
+                    if debug:
+                        logging.info(
+                            f"is_cache:{api['cache']} | status_code:{api['status']} | url:{api['url']} | took:{api['took']} seconds")
+                    return api["reparse"]
 
                 else:
-                    if resp.from_cache:
-                        if debug:
-                            logging.info(f"Not hitting the API, using cache")
-                        res = await resp.json()
-                        return res
-
-                    elif resp.from_cache is False and len(api_hit) < 3:
-                        if debug:
-                            logging.info(f"API hit {len(api_hit) + 1}")
-                        api_hit.append(1)
-                        res = await resp.json()
-                        await sleep(Jikan.simulate_hit)
-                        ## print(f"ratelimit hit 1: {len(api_hit)}")
-                        return res
-
-                    else:
-                        if debug:
-                            logging.info(
-                                f"API hit exceeded 3, Reseting hit to 1 and add delay"
-                            )
-                        api_hit.clear()
-                        api_hit.append(1)
-                        res = await resp.json()
-                        await sleep(1)
-                        ## print(f"ratelimit hit 2: {len(api_hit)}")
-                        return res
-
-            except Exception as e:
-                raise Exception(
-                    f"Failed to get data from: {path} with: {someid} due to: {e}"
-                )
+                    api_hit.clear()
+                    if debug:
+                        logging.info(
+                            f"Third conditions of request, apply sleep for {Jikan.constant_hit} seconds..")
+                    await asyncio.sleep(Jikan.constant_hit)
+                    api_hit.append(1)
+                    if debug:
+                        logging.info(
+                            f"Third should back to first condtions, API hit {len(api_hit)}")
+
+                    api = await fetch_hit(
+                        cache=sequel_cfg,
+                        endpoint=f"{api}/{rm_slash(endpoint)}",
+                        headers=ua,
+                    )
+                    if debug:
+                        logging.info(
+                            f"is_cache:{api['cache']} | status_code:{api['status']} | url:{api['url']} | took:{api['took']} seconds")
+
+                    return api["reparse"]