rss-feed

RSS Feed Mapping, Scraper, and Publishing Tools

This repo owns the source/data layer for the novel announcement system.

It stores novel and host metadata, generates RSS feeds, checks host updates, sends a few direct Discord reports/tools, and provides the installable mapping package used by the Discord announcement repos.

The repo now uses split TOML mapping files instead of keeping all novel data directly inside novel_mappings.py.

Repository Structure

Important files and folders:

rss-feed/
├─ .github/workflows/
│  ├─ update_free_feed.yml
│  ├─ update_paid_feed.yml
│  ├─ update_comments.yml
│  ├─ create_novel_toml.yml
│  ├─ update_novel_card.yml
│  ├─ publish_novel_card.yml
│  ├─ publish_membership_update.yml
│  ├─ monthly_revenue.yml
│  ├─ nu_weekly_readers.yml
│  └─ send_token_alert.yml
├─ novel_mappings.py
├─ mappings/
│  ├─ __init__.py
│  ├─ output_feeds.toml
│  ├─ hosts/
│  │  └─ mistmint_haven.toml
│  └─ novels/
│     ├─ amlwc.toml
│     ├─ atvhe.toml
│     ├─ ec.toml
│     ├─ hiaflg.toml
│     ├─ tdlbkgc.toml
│     ├─ tvitpa.toml
│     └─ wsmsc.toml
├─ host_utils/
│  ├─ __init__.py
│  ├─ host_dragonholic.py
│  ├─ host_nu_comments.py
│  ├─ host_titv.py
│  └─ mistmint_haven/
│     ├─ __init__.py
│     ├─ common.py
│     ├─ client.py
│     ├─ free_chapters.py
│     ├─ paid_chapters.py
│     └─ comments.py
├─ feed_common.py
├─ free_feed_generator.py
├─ paid_feed_generator.py
├─ comments.py
├─ message_renderer.py
├─ message_templates/
│  ├─ membership_update.toml
│  ├─ nu_weekly_readers.toml
│  ├─ publish_novel_card.toml
│  ├─ revenue_report.toml
│  └─ token_alert.toml
├─ novelupdates/
│  ├─ nu_weekly_readers.py
│  └─ nu_readers.json
├─ revenue/
│  ├─ report.py
│  ├─ state.json
│  └─ hosts/
├─ token/
│  ├─ send_token_alert.py
│  └─ token_alert_state.json
├─ tools/
│  ├─ audit_dead_host_utils.py
│  ├─ create_novel_toml.py
│  ├─ publish_membership_update.py
│  ├─ publish_novel_card.py
│  └─ update_novel_card.py
├─ free_chapters_feed.xml
├─ paid_chapters_feed.xml
├─ aggregated_comments_feed.xml
├─ novel_card_targets.json
├─ requirements.txt
├─ pyproject.toml
└─ README.md

What This Repo Produces

Generated RSS files:

free_chapters_feed.xml
paid_chapters_feed.xml
aggregated_comments_feed.xml

These are read by:

discord-webhook
mistmint-discord

The RSS files are the bridge between scraping/API logic and Discord announcements.

Discord Integrations, Routes, and Mentions

External Discord repos are configured in:

config/integrations.json

The important split is:

primary_discord
→ your private/master Discord target; normally discord-webhook; receives announcements for all novels

host_discord_targets
→ optional extra host/server-specific targets, such as mistmint-discord or a future host Discord

A host-specific Discord target is an extra destination, not a replacement for primary_discord. If a novel’s host has no entry in host_discord_targets, the direct announcement tools still post to the primary Discord only.

Example thread-based host target:

{
  "mistmint_discord": {
    "raw_base": "https://raw.githubusercontent.com/Cannibal-Turtle/mistmint-discord/main",
    "paths": {
      "server_json": "config/server.json",
      "thread_id_map": "config/thread_id_map.json"
    }
  },
  "host_discord_targets": {
    "mistmint_haven": {
      "integration": "mistmint_discord",
      "routes": {
        "membership_update": {
          "type": "thread_map",
          "map_key": "thread_id_map",
          "default_path": "config/thread_id_map.json"
        },
        "special_announcement": {
          "type": "thread_map",
          "map_key": "thread_id_map",
          "default_path": "config/thread_id_map.json"
        }
      }
    }
  }
}

Example channel-based host target:

{
  "wuxiaworld_discord": {
    "raw_base": "https://raw.githubusercontent.com/Cannibal-Turtle/wuxiaworld-discord/main",
    "paths": {
      "server_json": "config/server.json",
      "roles_json": "config/roles.json",
      "novel_discord_map": "config/novel_discord_map.toml",
      "state": "state.json"
    }
  },
  "host_discord_targets": {
    "wuxiaworld": {
      "integration": "wuxiaworld_discord",
      "routes": {
        "membership_update": {
          "type": "channel",
          "channel_key": "membership_updates"
        },
        "special_announcement": {
          "type": "channel",
          "channel_key": "announcements"
        }
      }
    }
  }
}

For channel routes, channel_key points to a key in that Discord repo’s config/server.json. New Discord repos may use either flat channel IDs:

{
  "guild_id": "123456789012345678",
  "global_mention": "<@&444444444444444444>",
  "announcements": "222222222222222222",
  "membership_updates": "333333333333333333"
}

or nested channel IDs:

{
  "guild_id": "123456789012345678",
  "global_mention": "<@&444444444444444444>",
  "channels": {
    "announcements": "222222222222222222",
    "membership_updates": "333333333333333333"
  }
}

For thread routes, map_key points to a path entry in that integration’s paths, usually thread_id_map, and the tool looks up the novel short_code in that JSON file. Missing host-specific thread/channel targets should mean “no extra host post,” not “skip the primary Discord post.”

Mention Resolution

For each target Discord integration, mention data is loaded from that target repo, not from the novel host mapping:

load config/server.json
→ global_mention

load config/roles.json
→ complete / ongoing / nsfw / membership / special_announcement / etc.

load config/novel_discord_map.toml
→ per-novel role / emoji / role URL

then the event decides which pieces to include

Recommended ownership:

File in target Discord repo Owns
config/server.json guild_id, channel IDs, raw global_mention such as @everyone, @here, <@USER_ID>, or <@&ROLE_ID>
config/roles.json named role IDs such as complete, ongoing, nsfw, membership, membership_update, special, special_announcement
config/novel_discord_map.toml per-novel Discord role/emoji/role URL keyed by novel short code
state.json completion state used to choose complete vs ongoing status role

Event examples:

membership_update
→ global_mention + membership/membership_update role + maybe novel role + status/nsfw if configured by the tool

special_announcement
→ global_mention + special/special_announcement role + maybe novel role + status/nsfw if configured by the tool

completed_novel
→ novel role + complete role + maybe global_mention

regular chapter
→ novel role + nsfw role if needed + maybe global_mention

In this repo, direct manual announcement mention strings are assembled by:

Script Responsible functions
tools/publish_membership_update.py resolve_global_mention(), resolve_event_role_mention(), resolve_novel_role_mention(), resolve_status_role_id(), build_global_mention(), allowed_mentions_for_mention()
tools/publish_special_announcement.py same pattern as membership: it resolves global/event/novel/status roles, builds the mention string, and sets allowed_mentions

config_loader.py only loads generic integration data. It should not grow new host-specific helpers for every future server.

Regular chapter announcements and completed-novel announcements are mostly assembled inside the downstream Discord repos that consume the RSS feeds, such as discord-webhook and mistmint-discord.

Mapping Files

novel_mappings.py

novel_mappings.py is the loader/front door.

Other scripts and repos can still import:

from novel_mappings import HOSTING_SITE_DATA

This keeps dependent scripts working while the actual editable data lives under:

mappings/

novel_mappings.py loads:

mappings/output_feeds.toml
mappings/hosts/*.toml
mappings/novels/*.toml

and builds HOSTING_SITE_DATA automatically.

Output Feed Config

mappings/output_feeds.toml

This file stores generated RSS feed URLs.

These are global repo-level feeds, not host-specific feeds.

free_feed = "https://raw.githubusercontent.com/Cannibal-Turtle/rss-feed/main/free_chapters_feed.xml"
paid_feed = "https://raw.githubusercontent.com/Cannibal-Turtle/rss-feed/main/paid_chapters_feed.xml"
comments_feed = "https://raw.githubusercontent.com/Cannibal-Turtle/rss-feed/main/aggregated_comments_feed.xml"

Novel TOML files use flags like:

has_free = true
has_paid = true
has_comments = true

Then novel_mappings.py injects the correct feed URLs automatically.

Host Mapping Files

Host-level data lives in:

mappings/hosts/

Example:

mappings/hosts/mistmint_haven.toml

Example structure:

host = "Mistmint Haven"

translator = "Cannibal Turtle"
translator_url = "https://example.com/account/@CannibalTurtle"
host_logo = "https://example.com/logo.png"
coin_emoji = "🪙"
ticket_emoji = "🎟️"

free_feed_url = "https://example.com/feed/"
novel_free_feed_url = "https://example.com/novels/{slug}/feed"
paid_feed_url = "https://example.com/feed/paid/"
novel_paid_feed_url = "https://example.com/novels/{slug}/paid-feed"
chapters_api_url = "https://api.example.com/api/novels/slug/{slug}/chapters"

free_chapters_source = "feed"
paid_chapters_source = "api"
chapter_mode = "auto"
comments_api_url = "https://api.example.com/..."
novels_api_url = "https://api.example.com/api/my-novels"

# Comment source modes:
# "trans"  = use tokened author dashboard endpoint; best metadata/reply tracking, token required
# "public" = use no-token /comments/novel/{identifier}; novel-page comments only, no chapter thread lookup
# "auto"   = try "trans" first; if token is missing/expired, fall back to "public"
comments_source = "auto"

# Name of the GitHub secret that stores this host's login token/cookie.
token_secret = "MISTMINT_COOKIE"

Host files should contain data shared by all novels on that host.

Examples:

Do not put per-novel data here.

Novel Mapping Files

Novel-level data lives in:

mappings/novels/

Each novel gets its own TOML file.

Example:

mappings/novels/amlwc.toml

Example:

host = "Mistmint Haven"
title = "After the Male Leads Went Crazy, They All Turned into Male Ghosts"
short_code = "AMLWC"

novelupdates_url = "https://www.novelupdates.com/series/after-the-male-leads-went-crazy-they-all-turned-into-male-ghosts"
novel_url = "https://www.mistminthaven.com/novels/after-the-male-leads-went-crazy-they-all-turned-into-male-ghosts"
featured_image = "https://web-novel-mistmint.s3.ap-southeast-1.amazonaws.com/novels/example-cover.jpg"
novel_id = "4221504f-49cd-4c8b-9c98-89e8b67705df"

chapter_count = "93 Chapters"
last_chapter = "Chapter 93"
start_date = "20/6/2026"
has_free = true
has_paid = true
is_nsfw = false
is_membership = false

discord_color = "#c90016"

tags = ["chinese", "quick transmigration", "supernatural"]
site_genres = ["Horror", "Supernatural", "Transmigration"]
history_file = "arc_history/amlwc_history.json"

custom_description = """
Optional multiline description here.

TOML supports triple-quoted multiline strings, so summaries are easier to paste and edit than JSON.
"""

Important Novel Fields

Required Fields

Field Purpose
host Must match a host file, e.g. "Mistmint Haven"
title Novel title used in feeds and status matching
short_code Stable short code used across bots and workflows
novel_url Main novel page
featured_image Cover image URL
novel_id Host/API novel ID when available; important for Mistmint API/comment tools
has_free Whether this novel appears in the free feed
has_paid Whether this novel appears in the paid feed
is_nsfw Whether this novel should be categorized as NSFW
is_membership Whether this novel is currently membership-only/available for membership

Optional Fields

Field Purpose
novelupdates_url Novel Updates page URL
chapter_count Display text for completion/status cards
last_chapter Completion checker target
start_date Used to calculate “After X of updates…” in completion messages
has_comments Comments feed flag; defaults to true unless explicitly set to false
tags Discord-supported genre tags, such as chinese, modern, romance, bl; downstream Discord repos use this for role mentions
site_genres Full original Mistmint Haven genre names from the API, kept for reference even if some are not Discord-supported tags
history_file Arc checker history file
discord_color Novel-specific embed color for Discord repos
theme_color Optional alternate novel color field
custom_description Multiline description for manual publishing/status cards

Empty Optional Fields

Use empty strings instead of deleting optional fields when you want scripts to safely skip related behavior.

start_date = ""
discord_color = ""
site_genres = []
history_file = ""

Meaning:

Field Empty Behavior
start_date = "" Completion announcement omits the duration phrase
discord_color = "" Discord repos use their normal/default color logic
site_genres = [] No Mistmint host genres are stored
history_file = "" Arc checker skips arc tracking for the novel

NSFW and Membership Tracking

NSFW

NSFW is controlled in the novel TOML:

is_nsfw = true

This can flow into RSS categories and Discord announcements.

Membership

Membership status is controlled in the novel TOML:

is_membership = true

The manual membership tool can update this automatically when a novel is announced as membership-available. Banners for membership are automatically blurred if novel is nsfw.

Helper Functions from novel_mappings.py

Downstream repos can use:

from novel_mappings import HOSTING_SITE_DATA

and helpers such as:

get_novel_details_by_short_code(short_code)
find_novel_by_short_code(short_code)
short_code_has_free_chapters(short_code)
short_code_has_paid_chapters(short_code)
short_code_has_comments_feed(short_code)
resolve_short_code(title, host)

Short codes are the stable bridge between this repo and the Discord repos.

Installable Package

pyproject.toml makes this repo installable as:

cannibal-turtle-rss-feed

Install from GitHub:

pip install --upgrade git+https://github.com/Cannibal-Turtle/rss-feed.git@main

The package currently installs:

novel_mappings.py
mappings/

including:

mappings/hosts/*.toml
mappings/novels/*.toml
mappings/output_feeds.toml

It is mainly a shared mapping package for the Discord repos.

The scraper/feed engine files such as host_utils/, tools/, and feed generator scripts are part of the repo, but are not currently packaged by pyproject.toml unless packaging is expanded.

Requirements

Install local script dependencies with:

pip install -r requirements.txt

Current main dependencies:

feedparser
PyRSS2Gen
aiohttp
beautifulsoup4
requests

Some workflows/scripts may also install:

discord.py
python-dateutil
tomli

tomli is only needed below Python 3.11.

Host Utilities

Host-specific scraper/API logic lives in:

host_utils/

Current shape:

host_utils/
├─ __init__.py
├─ host_dragonholic.py
├─ host_nu_comments.py
├─ host_titv.py
└─ mistmint_haven/
   ├─ __init__.py
   ├─ common.py
   ├─ client.py
   ├─ free_chapters.py
   ├─ paid_chapters.py
   └─ comments.py

host_utils/__init__.py is the host registry/dispatcher.

Use:

from host_utils import get_host_utils

utils = get_host_utils("Mistmint Haven")

The Mistmint implementation is split into:

File Purpose
common.py Shared helpers, parsing, diagnostics, settings, state helpers
client.py Mistmint API/client helpers
free_chapters.py Free chapter feed/API logic
paid_chapters.py Paid chapter scraping/API/update logic
comments.py Comments, replies, sticker/comment link logic
__init__.py Assembles MISTMINT_UTILS

The host registry should lazy-load hosts so one host does not require another host’s dependencies at import time.

RSS Feed Generators

The chapter generators are separated by chapter type, not by source method:

free_feed_generator.py
→ builds free_chapters_feed.xml
→ formats free/public chapter items

paid_feed_generator.py
→ builds paid_chapters_feed.xml
→ formats paid/premium chapter items, including paid history and coin/price fields

Shared generator rules live in:

feed_common.py

feed_common.py owns shared helpers such as:

Source Scope Rule

The generators treat scope separately from chapter type.

host/global feed
→ fetch once
→ scan entries
→ match entry title to mapped novel
→ no completion gate

novel-level feed
→ loop novels
→ check completion state before fetching that novel feed
→ parse entries for that novel

novel-level API
→ loop novels
→ check completion state before API request
→ fetch chapter data
→ filter by chapter type

This keeps host/global feeds simple: if the feed has an entry, the generator includes it. Completion state is only a fetch-saving gate for novel-scoped sources.

Source Config

Host TOML controls which source method each chapter type uses:

free_chapters_source = "feed"
paid_chapters_source = "api"

The source method is not the same as chapter type.

For example, Mistmint’s chapter API returns all chapters. The free and paid logic filter the same chapter data differently:

free API logic
→ keep chapters where isFree is true

paid API logic
→ keep chapters where isFree is false and the chapter is not hidden

A host/global feed is recognized when a feed URL is defined in mappings/hosts/*.toml, such as:

free_feed_url = "https://www.mistminthaven.com/feed/"

A host can also define reusable per-novel feed templates alongside its global feeds:

novel_free_feed_url = "https://www.mistminthaven.com/novels/{slug}/feed"
novel_paid_feed_url = "https://www.mistminthaven.com/novels/{slug}/paid-feed"

These do not replace the global feeds. In feed_api mode the matching per-novel feed is the first fallback when its global feed is capped or unavailable. A host only needs to define the URLs it actually provides.

A novel-level feed is recognized when a feed URL is defined in a specific mappings/novels/*.toml file.

A URL template like this is stored at host level but fetched per novel because it needs the novel slug:

chapters_api_url = "https://api.mistminthaven.com/api/novels/slug/{slug}/chapters"

Feed/API Fetch Behavior

Simple rule:

global feed URL = one async fetch
slugged / per-novel feed URL = async concurrent fetches
paid / free API source = async concurrent scraping

Examples:

free_feed_url = "https://www.mysite.com/feed/"
paid_feed_url = "https://www.mysite.com/feed/premium/"

These are global feeds, so each one is fetched once.

free_feed_url = "https://www.mysite.com/feed/{novel_slug}/"
paid_feed_url = "https://www.mysite.com/feed/premium/{novel_slug}/"

These are per-novel feeds, so the generator creates async tasks per novel and fetches them concurrently.

A host may expose both scopes at once:

free_feed_url = "https://www.mysite.com/feed/"
novel_free_feed_url = "https://www.mysite.com/novels/{slug}/feed"
paid_feed_url = "https://www.mysite.com/feed/paid/"
novel_paid_feed_url = "https://www.mysite.com/novels/{slug}/paid-feed"
chapters_api_url = "https://api.mysite.com/novels/{slug}/chapters"

With either chapter source set to feed_api, its generator uses the same scope-aware chain:

host/global free or paid feed
→ if capped or unavailable, fetch mapped novels through the matching per-novel feed template
→ use chapters_api_url only for novels whose per-novel feed is missing, failed, or also looks capped
→ de-duplicate by GUID/link

The free and paid generators each write a temporary fallback report for the Discord alert step. The report identifies whether the current batch used per-novel feeds, API fallback, or both; it is not committed to the repository.

For shared free + paid feeds:

free_feed_url = "https://www.mysite.com/feed/{novel_slug}/"
paid_feed_url = "https://www.mysite.com/feed/{novel_slug}/"

The generator fetches the same feed for free and paid, but host_utils must know how to separate free entries from paid entries. Host-specific free/paid detection belongs in host_utils/<host_name>/.

Shared generator helpers belong in feed_common.py.

free_feed_generator.py

Builds:

free_chapters_feed.xml

Uses novel TOML entries with:

has_free = true

Supports:

host/global free feed
novel-level free feed
host utility/API loader, such as Mistmint free API mode

The XML item structure is kept inside this generator so the free RSS output format stays stable.

Builds:

paid_chapters_feed.xml

Uses novel TOML entries with:

has_paid = true

Supports:

host/global paid feed
per-novel paid feed as a primary source or global-feed fallback
novel-level paid API/scraper fallback for uncovered, failed, or capped novel feeds
manual/state fallback when a host uses it

When paid_chapters_source = "feed_api" and both feed scopes are configured, the paid generator follows global paid feed → per-novel paid feed → API, the same way the free generator handles its sources.

Paid-only behavior stays inside this generator, including:

paid_history.json + paid GUID handling (for manual entry only)
coin/price fields
paid-specific item formatting

comments.py

Builds:

aggregated_comments_feed.xml

Uses novel TOML entries with:

has_comments = true

The comments generator already asks novel_mappings.py for comment source URLs with novel fallback, so it can use host-level or novel-level comment API/feed config depending on the mapping.

Sample RSS Item Fields

Generated RSS items may include:

title
volume
chapter
chaptername
link
description
category
translator
short_code
featured_image_url
pub_date
host
host_logo_url
guid
guid_is_permalink

These are consumed by the Discord repos.

Feed Sorting

Paid/free feed sorting should not depend on TOML insertion order.

Where possible, sorting should use parsed dates, chapter numbers, and stable tie-breakers such as alphabetical title/short code.

Mistmint Haven Quick Setup

Modes

Mistmint host config can control source modes:

# Chapter source modes:
# "feed" = use a host-provided feed/RSS-style source for this chapter type
# "api"  = use the host chapter API/data source, then filter by chapter type
free_chapters_source = "feed"
paid_chapters_source = "api"

# Paid chapter fallback mode:
# "auto"   = try API/chapter data first; cookie is optional if the endpoint allows it
# "manual" = force mistmint_state.json/manual paid chapter fallback instead of API
chapter_mode = "auto"

# Comment source modes:
# "trans"  = use tokened author dashboard endpoint; best metadata/reply tracking, token required
# "public" = use no-token /comments/novel/{identifier}; novel-page comments only, no chapter thread lookup
# "auto"   = try "trans" first; if token is missing/expired, fall back to "public"
comments_source = "auto"

Typical meaning:

Setting Purpose
free_chapters_source Whether free chapters come from a feed-style source or the chapter API/data source
paid_chapters_source Whether paid chapters come from a feed-style source or the chapter API/data source
chapter_mode Mistmint paid fallback mode; auto tries API/chapter data, manual forces manual/state fallback
comments_source = "trans" Uses comments_api_url / comments/trans/all-comments; best metadata and reply tracking, but token/cookie is required
comments_source = "public" Uses public no-token /comments/novel/{identifier} endpoints. This is a novel-page comments fallback only: it does not resolve chapterId, does not call chapter comment endpoints, and should not be treated as a full chapter comments feed.
comments_source = "auto" Tries trans first, then falls back to public mode if the token is missing/expired

The public Mistmint comments endpoint is internal host logic, not user-facing repo config. The code builds it from BASE_API as /comments/novel/{identifier} and tries the mapped novel_id first, then the novel slug. Based on the current public API shape, this endpoint returns comments attached to the novel itself, not full chapter comment threads. Keep this in Python unless Mistmint changes endpoint structure often enough that it becomes worth exposing a separate URL template.

Public comments mode is intentionally limited. It does not resolve chapterId, it does not call /comments/chapter/{chapterId}, and it does not build reply chains. Use trans mode for the complete author-wide comments feed and richer reply tracking.

Mapping

For each Mistmint novel, the novel TOML should include:

host = "Mistmint Haven"
short_code = "CODE"
novel_url = "https://www.mistminthaven.com/..."
has_free = true
has_paid = true
has_comments = true

Required Repo Secrets

Mistmint API/private data may need:

Secret Purpose
MISTMINT_COOKIE Mistmint login/session cookie or token, depending on script
DISCORD_BOT_TOKEN Required for direct Discord tools/reports
GH_PAT Used when dispatching workflows or editing external repo files where needed

tools/create_novel_toml.py uses the host config’s token_secret first. For Mistmint, that normally means MISTMINT_COOKIE. It can also use MISTMINT_TOKEN if you provide bearer-token auth instead.

Token-Expiry Alerts

Token alert logic lives in:

token/send_token_alert.py
token/token_alert_state.json
message_templates/token_alert.toml

Template-specific user settings live in:

[settings]
global_mention = "||<@&1329392448798982214>||"

Token alert checks are host-agnostic. Any host file can opt in by defining a token secret:

token_secret = "HOST_COOKIE_OR_TOKEN_SECRET"

Optional host setting:

token_alerts = "auto"

Supported values:

Value Meaning
"auto" or omitted Alert for hosts with token_secret, except when comments_source = "public"
true Always alert when token_secret is present
false Never send token alerts for this host

Examples:

# Token is needed for private/API comment fetching. Alerts run.
token_secret = "MISTMINT_COOKIE"
comments_source = "auto"
token_alerts = "auto"

# Comments are intentionally public-only. Alerts are skipped in auto mode.
token_secret = "SOME_OPTIONAL_COOKIE"
comments_source = "public"
token_alerts = "auto"

# Token is used by another script even if comments are public. Alerts still run.
token_secret = "HOST_API_TOKEN"
comments_source = "public"
token_alerts = true

token-expiring alerts only work for JWT-style tokens/cookies with an exp field. token-invalid alerts are sent when a host comments loader raises an AUTH_ERROR, and follow the same token_alerts host policy.

The workflow:

.github/workflows/send_token_alert.yml

can be triggered by dispatch or manually.

Message Templates in rss-feed

This repo has direct Discord tools/reports, so it also has templates:

message_templates/membership_update.toml
message_templates/nu_weekly_readers.toml
message_templates/publish_novel_card.toml
message_templates/revenue_report.toml
message_templates/token_alert.toml

Unlike discord-webhook, this repo does not use config/embeds.json.

Template-specific user/repo settings should live in each TOML file under:

[settings]

Examples:

[settings]
global_mention = "||<@&1329392448798982214>||"
embed_color = "2D3F51"
novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"

Python should read these settings with:

load_template_settings("template_name")

This keeps fork-specific IDs, colors, and URLs out of Python.

Novel Updates Weekly Readers

Script:

novelupdates/nu_weekly_readers.py

Template:

message_templates/nu_weekly_readers.toml

State:

novelupdates/nu_readers.json

Workflow:

.github/workflows/nu_weekly_readers.yml

Purpose:

Template settings include:

[settings]
global_mention = "||<@&1329392448798982214>||"
embed_color = "2D3F51"
novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"
allow_role_pings = true
no_data_text = "_No data this week (no NU counts retrieved)._"

The novel_discord_map_url lets this repo resolve novel short codes to Discord role mentions without storing Discord role data inside rss-feed mappings.

Monthly Revenue Report

Script:

revenue/report.py

Template:

message_templates/revenue_report.toml

State:

revenue/state.json

Workflow:

.github/workflows/monthly_revenue.yml

Purpose:

Template settings include:

[settings]
global_mention = "||<@&1329392448798982214>||"
embed_color = "C9D3FF"
novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"

Revenue rows are controlled by template blocks such as:

[row_basic]
[row_membership]
[first_run]
[month_header]
[monthly_total]
[empty_report]

Create Novel TOML Tool

Script:

tools/create_novel_toml.py

Workflow:

.github/workflows/create_novel_toml.yml

Purpose:

For Mistmint Haven, the host file must include:

novels_api_url = "https://api.mistminthaven.com/api/my-novels"
token_secret = "MISTMINT_COOKIE"

Workflow inputs:

Input Required? Purpose
host Yes Hosting site, e.g. Mistmint Haven
title Yes Novel title exactly/as shown in the host dashboard
short_code Yes New short code, e.g. AMLWC
chapter_count No Optional display text, e.g. 93 Chapters; blank writes ""
last_chapter No Optional target text, e.g. Chapter 93; blank writes ""
discord_color No Optional hex color, e.g. #c90016; blank writes ""
quick_transmigration No Checkbox. Tick this only when the novel is quick transmigration; adds quick transmigration to tags.
infinite_flow No Checkbox. Tick this only when the novel is infinite flow; adds infinite flow to tags.
has_arcs Yes If true, creates arc_history/<short_code>_history.json and sets history_file
dry_run Yes If true, previews the TOML in the Actions log without committing
overwrite Yes If true, allows replacing an existing mappings/novels/<short_code>.toml

Recommended first run:

dry_run = true
overwrite = false

Then rerun with:

dry_run = false

once the generated TOML looks right.

Tags and Mistmint genres

tags should only contain tags that exist in the Discord repo’s config/tag_roles.json, because downstream Discord repos use tags for role mentions.

site_genres stores the full original genre names from the Mistmint API. It does not need to match Discord role tags and is kept as a reference copy of what Mistmint lists on the novel.

World-hopping uses two Actions checkboxes. Leave both unchecked when the novel is not quick transmigration or infinite flow. If it is world-hopping, tick exactly one checkbox.

When a world-hopping checkbox is ticked, the selected tag is written directly into tags:

tags = ["chinese", "quick transmigration", "modern", "romance", "bl"]
site_genres = ["Modern", "Romance", "Yaoi", "Transmigration"]

If quick transmigration or infinite flow is selected, the tool removes plain transmigration from tags automatically. This keeps a world-hopping novel from ending up with both the broad transmigration role and the specific world-hopping role.

Leaving both world-hopping checkboxes unchecked writes normal tags only:

tags = ["chinese", "transmigration", "modern", "romance", "bl"]
site_genres = ["Modern", "Romance", "Yaoi", "Transmigration"]

Automatic Novel Status Updater

Script:

tools/update_novel_card.py

State/targets:

novel_card_targets.json

Workflow:

.github/workflows/update_novel_card.yml

Purpose:

Workflow inputs:

title: "Novel title"
host: "Mistmint Haven"

novel_card_targets.json stores Discord message targets by short code.

Manual Novel Card Tool

Script:

tools/publish_novel_card.py

Template:

message_templates/publish_novel_card.toml

Workflow:

.github/workflows/publish_novel_card.yml

Workflow inputs:

short_code: "AMLWC"
channel_id: "optional extra Discord channel/thread ID"

What it does:

Template settings include:

[settings]
archive_channel_id = "1463476725253144751"
novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"

Membership Update Tool

Script:

tools/publish_membership_update.py

Template:

message_templates/membership_update.toml

Workflow:

.github/workflows/publish_membership_update.yml

Workflow inputs:

short_code: "AMLWC"
banner_url: "https://..."

What it does:

Template settings can still include a fallback public mention:

[settings]
public_global_mention = "||@everyone||"

That setting is the last fallback. Preferred global/public pings should live in the target Discord repo’s config/server.json as global_mention.

Banner cropping for membership updates is configured in message_templates/membership_update.toml:

[settings.banner]
ratio = "4:1"  # use "original" to preserve the full featured image

The workflow’s optional banner_ratio input can override this per run. Ratio values such as 8:3 crop/resize to that shape; original performs no crop or resize. The same controls are available for special announcements in message_templates/special_announcement.toml and publish_special_announcement.yml. With crop_position = "auto", a lightweight Pillow-only heuristic compares text-like horizontal activity near the top and bottom: stronger top activity selects lower center, stronger bottom activity selects upper center, and unclear results default to upper center. Manual crop positions remain available.

Workflow Inputs

create_novel_toml.yml

Input Purpose
host Hosting site, e.g. Mistmint Haven
title Novel title exactly/as shown in host dashboard
short_code Novel short code, e.g. AMLWC
chapter_count Optional chapter count text
last_chapter Optional last-chapter text
discord_color Optional novel embed color
quick_transmigration Whether to add quick transmigration to tags
infinite_flow Whether to add infinite flow to tags
has_arcs Whether to create an arc history file
dry_run Preview without committing
overwrite Allow replacing an existing mapping file

publish_novel_card.yml

Input Purpose
short_code Novel short code, e.g. AMLWC
channel_id Optional extra Discord channel/thread ID

publish_membership_update.yml

Input Purpose
short_code Novel short code, e.g. AMLWC
banner_url Optional finished banner URL
banner_ratio Optional ratio override such as 8:3, or original for no crop/resize
crop_position Auto or manual crop position when using a ratio

publish_special_announcement.yml

Input Purpose
short_code Novel short code
banner_url Optional finished banner URL
banner_ratio Optional ratio override such as 8:3, or original for no crop/resize
crop_position Auto or manual crop position when using a ratio

update_novel_card.yml

Input Purpose
title Novel title
host Hosting site, e.g. Mistmint Haven

Workflows

Workflow Purpose
update_free_feed.yml Regenerates free RSS feed, scheduled daily
update_paid_feed.yml Regenerates paid RSS feed, scheduled hourly
update_comments.yml Regenerates comments RSS feed, scheduled hourly
create_novel_toml.yml Creates a new novel TOML from a configured host API
update_novel_card.yml Edits existing Discord novel status cards
publish_novel_card.yml Manually posts a novel/status card
publish_membership_update.yml Manually posts membership announcement
publish_special_announcement.yml Manually posts a special announcement
monthly_revenue.yml Posts monthly revenue report
nu_weekly_readers.yml Posts weekly NU reader-count report
send_token_alert.yml Sends token warning/error alerts

Adding a New Novel on an Existing Host

Preferred method for a configured API host:

  1. Run:

    create_novel_toml.yml

  2. Start with:

    dry_run = true
overwrite = false

  3. Check the generated TOML in the Actions log.

  4. Rerun with:

    dry_run = false

  5. Confirm the new file exists in:

    mappings/novels/<short_code>.toml

Manual fallback:

  1. Create a novel TOML file in:

    mappings/novels/

  2. Add the core fields:

    host = "Mistmint Haven"
title = "Novel Title"
short_code = "CODE"

novelupdates_url = "https://www.novelupdates.com/series/novel-title"
novel_url = "https://www.mistminthaven.com/novels/novel-title"
featured_image = "https://..."
novel_id = ""

chapter_count = ""
last_chapter = ""
start_date = ""
has_free = true
has_paid = true
is_nsfw = false
is_membership = false

discord_color = ""

tags = ["chinese"]
site_genres = []
history_file = ""

custom_description = """
Description here.
"""

  3. If the novel has arcs, set:

    history_file = "arc_history/code_history.json"

    and create the matching JSON file with:

    {}

  4. Add Discord role/emoji/role URL data in the Discord repo that owns the announcement role map, usually:

    discord-webhook/config/novel_discord_map.toml

  5. If the novel’s host has an extra host-specific Discord target, add the required channel/thread mapping in that host Discord repo. For a thread target, that usually means:

    <host-discord-repo>/config/thread_id_map.json

    Channel IDs belong in that Discord repo’s config/server.json; config/integrations.json only stores which channel_key to read.

  6. Run the relevant feed workflow.

  7. Run publish_novel_card.yml if you need a manual card.

  8. Confirm novel_card_targets.json updates if the novel has a status card.

Adding a New Hosting Site

  1. Add host config:

    mappings/hosts/new_host.toml

  2. Add novel configs:

    mappings/novels/code.toml

  3. Decide the source scope/method for free and paid chapters.

    Host/global feed example:

    # mappings/hosts/new_host.toml
free_chapters_source = "feed"
free_feed_url = "https://example.com/feed/"

    Global feed with per-novel and API fallback:

    # mappings/hosts/new_host.toml
free_feed_url = "https://example.com/feed/"
novel_free_feed_url = "https://example.com/novels/{slug}/feed"
paid_feed_url = "https://example.com/feed/paid/"
novel_paid_feed_url = "https://example.com/novels/{slug}/paid-feed"
chapters_api_url = "https://api.example.com/novels/{slug}/chapters"

    Then set the relevant free_chapters_source and/or paid_chapters_source to feed_api in config/source_modes.json.

    Novel-level feed example:

    # mappings/novels/code.toml
free_feed_url = "https://example.com/novel/code/feed/"

    Novel-level API example:

    # mappings/hosts/new_host.toml
paid_chapters_source = "api"
chapters_api_url = "https://api.example.com/novels/{slug}/chapters"

  4. Add or update host utilities in:

    host_utils/

  5. Register the host in:

    host_utils/__init__.py

  6. Make sure feed generators can load the host utils.

  7. Add any required token/cookie secret name in the host TOML.

  8. Add Discord-side config only where needed:

    • discord-webhook is the primary/private Discord target and can hold all-novel role data.
    • A host-specific Discord repo is optional. Add one under host_discord_targets only if that host needs an extra channel/thread post.
    • If the host has no extra Discord target, direct announcement tools still post to the primary Discord.

Manual Publishing Checklist

When adding a new novel:

  1. Run create_novel_toml.yml, or manually create a novel TOML file in mappings/novels/.
  2. Add a unique short_code.
  3. Check tags, site_genres, chapter_count, last_chapter, and discord_color before publishing. For world-hopping, tick exactly one world-hopping checkbox so the matching role tag appears inside tags.
  4. Add Discord role/emoji/role URL data in the Discord repo.
  5. Run publish_novel_card.yml.
  6. Confirm novel_card_targets.json was updated.
  7. If the novel enters membership, run publish_membership_update.yml with:
    • short_code
    • banner_url

  8. Confirm the novel TOML now has:

    is_membership = true

Troubleshooting

ModuleNotFoundError: feedparser

Install requirements:

pip install -r requirements.txt

For workflows that import multiple scripts, also make sure the workflow installs needed dependencies before running Python.

Host import breaks an unrelated script

host_utils should lazy-load host modules.

A script asking for Mistmint should not import Dragonholic/TITV dependencies unless it requests those hosts.

TOML template placeholders render empty

A placeholder only works if Python passes the value in the render context.

Example:

color = "{embed_color}"

needs Python to pass:

{"embed_color": 0x2D3F51}

Template-specific defaults should live under:

[settings]

and Python should read them with load_template_settings(...).

Discord role/config URL should not be hardcoded in Python

Repo/user-specific IDs and URLs should live in template [settings], not in Python.

Good:

[settings]
global_mention = "||<@&1329392448798982214>||"
novel_discord_map_url = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/config/novel_discord_map.toml"

Completion state URL is missing

feed_common.py expects the canonical Discord completion state URL in:

[completion_state_url]
discord_webhook = "https://raw.githubusercontent.com/Cannibal-Turtle/discord-webhook/main/state.json"

If this URL is missing or unreachable, generators continue without completion skipping instead of failing the whole feed run.

JSON state file crashed

Empty files are invalid JSON.

Use:

{}

Design Guarantees

System Overview

Host sites / APIs
   ↓
rss-feed host utils + feed_common.py
   ↓
free / paid / comments RSS XML
   ↓
discord-webhook + mistmint-discord
   ↓
Discord announcements

Manual tools and reports also live in rss-feed:

NU weekly readers
monthly revenue
membership update
manual novel card
token alerts
status updater

Example Result

The final Discord status card is created once, then updated automatically when new free chapters are posted.

The status updater preserves the existing card layout and only edits the status field.

image