# goj: the whole design in charts Drawn from the code, not from the earlier design records: where these charts and `design.md` or `plan.md` disagree, the code and these charts are right. GitHub renders the `mermaid` blocks below. Rendered PNGs of all three charts live in `docs/img/` (`pipeline.png`, `dependencies.png`, `download-run.png`). Each was produced by copying its block below into a `.mmd` file and running `npx -y -p @mermaid-js/mermaid-cli mmdc -i .mmd -o .png -b white -s 2 -c render-config.json` against the installed Chrome, where `render-config.json` is `{"flowchart":{"useMaxWidth":false},"sequence":{"useMaxWidth":false}}` — without it mermaid squeezes a wide chart into an 800px viewport and the PNG comes out tiny. A standalone HTML rendering of chart 1 lives in `docs/pipeline.html`: the diagram inline as SVG with a Fit width / Actual size toggle, plus a legend of the boxes. `docs/pipeline-book.html` is a self-contained book that documents the pipeline part by part — one chapter per package, each with its own small diagram. SVG renders come from the same command with `-o .svg`; `-I ` gives each diagram a unique id so several can share one page without style collisions, and the book's inline copies have the embedded webfont stripped (one declaration in its head covers all of them). The chart-1 SVG also sits beside the PNGs as `docs/img/pipeline.svg`. Three views, from whole to one run: 1. [The whole pipeline](#1-the-whole-pipeline) — modes, packages, data flow. 2. [Package dependencies](#2-package-dependencies) — who may import whom. 3. [One download run](#3-one-download-run) — the same pipeline in time, with the concurrency. ## 1. The whole pipeline ```mermaid flowchart LR USER(["User"]) -->|"goj URL, or --list URL, or --download-list FILE"| FLAGS subgraph app["main.go — flags, wiring, exit codes (0 ok, 1 failed, 2 usage)"] FLAGS["flags: -o base, -c N, -n N, --no-progress, --list, --download-list,
the URL may sit before or after them"] FLAGS --> MODE{"mode"} MODE -->|"a page URL"| ONE["one(): download one video"] MODE -->|"--list URL"| LM["listMode(): crawl a listing page"] MODE -->|"--download-list FILE"| EACH["each(): batch of URLs, one file each"] EACH -->|"per URL, skipped when .mp4 or .ts is already on disk"| ONE end subgraph pagepkg["internal/page — HTML acquisition (utls) + URL extraction"] PCLIENT["Client.read: try browser TLS fingerprints in order,
ALPN pinned to http/1.1; never returns a half page"] PEX["Extract: first matching per-host Rule regex, gives hlsUrl"] PLX["ExtractLinks: ListingRules, gives video URLs + next-page URL"] end subgraph listingpkg["internal/listing — pagination walk + list file format"] CRAWL["Crawl: follow the next link page by page, dedupe videos first-seen,
stop on a repeat or the last page, cap 100 pages"] LFILE["Read / Write: one URL per line"] end subgraph hlspkg["internal/hls — what the playlist says (no I/O)"] PARSE["Parse: one #EXT-X-KEY (AES-128 + explicit IV),
ordered #EXTINF segments"] RESOLVE["Resolve: relative key / segment URIs against the playlist URL"] end subgraph mediapkg["internal/media — concurrent fetch, decrypt, in-order emit"] KEYF["fetchKey: exactly 16 bytes"] SEM["semaphore: bounds in-flight segment requests (default 32)"] WORK["one worker goroutine per segment:
GET, retry with backoff, 90s attempt timeout"] DEC["DecryptSegment: AES-128-CBC + PKCS#7 strip"] WIN["window: bounded reorder buffer, 4 x concurrency segments resident"] EMIT["emitter: the only writer of the output, strictly playlist order"] end subgraph outpkg["internal/output — the final file and its fallback"] SINK["Sink: append to .ts and tee the same bytes"] FFMPEG["ffmpeg -c copy, TS on pipe:0, writes .mp4.part"] MP4["rename to .mp4 on success, then delete the .ts"] KTS["ffmpeg absent: keep the joined .ts"] ABORT["Abort on failure: kill ffmpeg, remove partial .ts / .mp4.part"] end subgraph prog["internal/progress — live lines on stderr, only when a console"] SPIN["spinner: page / list phase, elapsed seconds"] BAR["bar: percent, done/total, bytes, average rate (no ETA)"] end SITE["jable.tv — behind a Cloudflare gate that decides on the TLS handshake"] MHOST["media host — speaks HTTP/1.1 only, serves playlist, key and segments"] LM --> CRAWL CRAWL --> PCLIENT CRAWL --> PLX CRAWL --> LFILE LM --> LISTOUT(["stdout: one video URL per line, plus wrote FILE.txt"]) LISTOUT -->|"the file feeds the third mode"| EACH ONE --> PCLIENT PCLIENT -.->|"HTTPS with a browser's ClientHello"| SITE PCLIENT --> PEX PEX -->|"media URL (m3u8)"| PARSE ONE -->|"GET playlist with Referer"| MHOST PARSE --> RESOLVE ONE --> SINK RESOLVE --> KEYF KEYF -->|"GET key"| MHOST RESOLVE --> WORK WORK -->|"GET segment"| MHOST WORK --> DEC DEC --> WIN SEM -.->|"gates"| WORK WIN --> EMIT EMIT -->|"ordered TS bytes"| SINK SINK --> FFMPEG SINK --> KTS FFMPEG --> MP4 EMIT -.->|"progress callback"| BAR PCLIENT -.-> SPIN CRAWL -.-> SPIN ONE -.->|"on failure"| ABORT classDef ext fill:#fdf0d5,stroke:#b8860b,color:#5a4200 class SITE,MHOST ext ``` ## 2. Package dependencies Solid arrows are imports; the leaves do no I/O of their own beyond the platform calls named. `internal/page` and `internal/media` are the two independent network stacks, and neither knows the other. ```mermaid flowchart LR main["main.go
flags, wiring, exit codes"] --> page["internal/page
page reads + URL extraction"] main --> hls["internal/hls
playlist parse, no I/O"] main --> listing["internal/listing
pagination walk + list file"] main --> media["internal/media
fetch, decrypt, ordered emit"] main --> output["internal/output
.ts sink + ffmpeg remux"] main --> progress["internal/progress
live terminal lines"] listing -->|"page reads"| page media -->|"playlist types"| hls page -->|"TLS fingerprints"| utls["refraction-networking/utls
the only module dependency"] output -.->|"os/exec, when present"| ffmpeg["ffmpeg
external tool"] ``` ## 3. One download run The whole-pipeline view in time, with the two concurrency structures that keep it fast and bounded. ```mermaid sequenceDiagram autonumber actor U as User participant M as main.go (one) participant P as page.Client (utls) participant H as hls participant D as media.Download participant F as segment workers participant W as window participant E as emitter participant O as output.Sink participant X as ffmpeg participant R as progress.Renderer Note over R: draws only when stderr is a console U->>M: goj https://jable.tv/videos/ipx-976/ M->>R: StartSpinner("page") M->>P: Get(ctx, pageURL) P->>P: fingerprint loop: Chrome/120, Chrome/131, Edge/106, Safari/16.0 P-->>M: page source, never a half page M->>R: StopSpinner() M->>M: page.Extract(src), giving hlsUrl M->>M: GET playlist with Referer jable.tv M->>H: Parse(body) H-->>M: Key (AES-128 + IV) and the ordered segments M->>O: NewSink(ctx, tsPath, base) Note over O,X: the Sink writes the .ts and tees bytes into ffmpeg's stdin,
and ffmpeg starts on the first byte M->>D: Download(ctx, pl, playlistURL, opts, sink) D->>D: fetchKey() returns 16 bytes loop every segment index D->>F: spawn worker i, admitted by window then semaphore F->>F: GET segment, retry with backoff on transient errors F->>F: DecryptSegment() AES-128-CBC, strip PKCS#7 F->>W: put(i, plaintext) end loop strict playlist order W->>E: segment next is ready E->>O: Write(ordered TS bytes) O->>X: tee to stdin E->>R: Progress(done, total, bytes) end D-->>M: Result{Bytes, Segments} M->>O: Finish() O->>X: close stdin, wait for exit X-->>O: exit 0 O->>O: renames .mp4.part to .mp4 and removes the .ts O-->>M: Product{Path, Remuxed} M->>U: done: ipx-976.mp4 Note over M,O: any failure: Sink.Abort() kills ffmpeg and removes the partial .ts and .mp4.part ```