The goj pipeline

1 Overview — one run end to end

Give goj a page URL. It reads the page through the site's TLS-fingerprint gate, pulls the media playlist's URL out of the page source, fetches the AES key and the segments, decrypts them, streams them in playlist order into an intermediate .ts, and tees every byte of that stream into a concurrent ffmpeg that produces a lossless MP4. Two sibling modes never download: --list crawls a listing page into a list of video URLs, and --download-list feeds such a list back in, downloading each URL.

[Z] ACTUAL SIZE

The whole pipeline in one picture. [Z] ACTUAL SIZE switches to the diagram's intrinsic width with a horizontal scrollbar.

A run, as it reports itself

page:   https://jable.tv/videos/ipx-976/
page     / 1s  waiting for the page
hls:    https://<media-host>/hls/<token>/<ts>/29000/29202/29202.m3u8
playlist: AES-128, 1906 segments
[##########--------------------]  52%   992/1906     1.37 GiB    38.2 MiB/s
write:  29202.ts (2691332468 bytes, 1906 segments)
remux:  ffmpeg -c copy -> ipx-976.mp4
done:   ipx-976.mp4

The spinner and the bar are one redrawing line on stderr; the rest is one line each on stdout. Both live lines appear only when stderr is a console.

The three modes

2 The CLI — main.go

main.go owns three things and nothing else: flags, wiring, and exit codes. It never touches the network, a playlist, or a byte of video; it hands work to the packages and prints what they report.

The surface

func run(args []string, stdout, stderr io.Writer) int
type downloader struct {
    client *http.Client       // the media client, HTTP/1.1 pinned
    bar    *progress.Renderer
    out    io.Writer          // stdout
    errOut io.Writer          // stderr
    conc   int                // -c, 0 means the default (32)
    maxSeg int                // -n, 0 means all
}
func (d downloader) one(ctx context.Context, pageURL, base string) error
func (d downloader) each(ctx context.Context, urls []string, dir string) []string
func listMode(ctx context.Context, pageURL, base string, noProgress bool, stdout, stderr io.Writer) int
func parseInterspersed(fs *flag.FlagSet, args []string) ([]string, error)
func slugOf(rawurl string) string

Flags

The URL may go before, between, or after the flags: parseInterspersed keeps parsing until the positional arguments run out, which the standard flag package does not do on its own.

Names and sanitising

• The output base defaults to the last path element of the URL (ipx-976 from /videos/ipx-976/).
• trimExt strips a known extension (.mp4 .ts .mkv .mov .webm .txt) from a caller-supplied -o, so -o sample.mp4 does not become sample.mp4.mp4.
• The intermediate .ts takes the playlist file's basename, so a run names it after its playlist rather than after -o.

The batch loop

• Reads the list with listing.Read — one URL per line, blank lines and duplicates tolerated.
• For each URL, if base.mp4 or base.ts already exists, the item is skipped and the run continues.
• One failure does not stop the batch: the error goes to stderr as it happens, the run carries on, and the exit status is 1 at the end.
• The closing line reports downloaded, skipped, failed counts.
• With -o, the directory is created first, and each item's name still comes from its own URL, so slugs stay unique.

Where the notes go

A single download prints its phases (page: hls: playlist: write: remux: done:) to stdout. --list inverts that: stdout carries exactly the URL list — so it pipes into anything — and every note (list:, videos:, wrote:) goes to stderr.

Exit codes

3 Reading the page — internal/page

The site sits behind a gate that decides on the TLS handshake itself. Go's own ClientHello is answered with the challenge under every header set tried, so this package presents a browser's ClientHello instead — and gets the page, in about a third of a second, with no JavaScript run and no window.

Why fingerprints, and why four

• The gate's verdict is per fingerprint and it moves as the gate is tuned, so a single one is not durable. Four are held and tried in order: Chrome/120, Chrome/131, Edge/106, Safari/16.0.
• Measured 2026-10-17: Chrome/120, Edge/106 and Safari/16.0 were answered with the page on every attempt; Chrome/131 on most; Chrome/133, Edge/85, Chrome/106, Firefox/120 and a latest-Chrome parrot with the challenge on every attempt.
• ALPN is pinned to http/1.1. The same fingerprints negotiate h2 happily, and over h2 the gate answers every one of them with the challenge, so the h2 layer's fingerprint must not be presented at all.

Reading is waiting on the rule

Get does not merely fetch: it waits until the source satisfies the same extraction rule that will later be applied, so the wait and the extraction can never disagree. List does the same with the listing rule. A half page is never returned; when every fingerprint is answered with the challenge, the error is ErrChallengeNotCleared.

Extraction is a table, not an if-chain

• Rules maps a host suffix to a regex whose capture group 1 is the media URL; the target's page carries it as an inline var hlsUrl = '…'.
• ListingRules is a second table for listing pages, which need a different shape: many video URLs plus a next-page link, against the single media URL behind no pagination.
• A site that changes how it embeds its URLs needs a new table entry, not new code.

type Source interface { Get(ctx context.Context, pageURL string) ([]byte, error) }
type Client struct { Timeout time.Duration }   // zero value is ready to use
func (c Client) Get(ctx context.Context, pageURL string) ([]byte, error)
func (c Client) List(ctx context.Context, pageURL string) ([]byte, error)
func Extract(pageURL string, src []byte) (string, error)
func ExtractLinks(pageURL string, src []byte) (videos []string, next string, err error)
var ErrChallengeNotCleared = errors.New("page: challenge not cleared")

The network-backed Client and the test Fake are the two implementations that earn the Source interface.

4 Crawling a listing — internal/listing

A model's or a category's page names many videos across more than one page. This package owns the walk, the dedupe, the stop, and the list file format. The browser work stays behind the Page interface; the production implementation is page.Client.

The walk

• The first URL gets a trailing slash, because without it the site answers 404; every later page comes from the site's own href and is resolved, never constructed.
• Videos are deduped by resolved URL in first-seen order — every card carries two anchors with the same href (thumbnail and title), and the dedupe collapses them.
• A page that points back at one already read ends the walk rather than looping.
• The walk stops on its own at the last page. maxPages = 100 only guards a site whose next link never repeats and never ends; reaching it is an error, not a silent truncation.
• The next link is matched by the 最後 » marker in the pagination control, and the pattern cannot match across a tag boundary: a changed marker yields no match — the walk stops honestly — rather than a wrong one.

The list file

Write emits one URL per line. Read keeps the first occurrence of each URL and drops blank lines, so a hand-edited or reordered file still works, and a URL that appears twice is downloaded once.

type Page interface { List(ctx context.Context, pageURL string) ([]byte, error) }
type Result struct { URLs []string; Pages int }
func Crawl(ctx context.Context, first string, p Page) (Result, error)
func Read(r io.Reader) ([]string, error)
func Write(w io.Writer, urls []string) error

5 What the playlist says — internal/hls

A pure model of an HLS media playlist: one optional playlist-level key, and the segments in playlist order. It performs no I/O — parsing bytes in, types out, URI resolution included.

What parses

• #EXTINF pairs a duration with the URI line that follows it, so slice order is playlist order.
• #EXT-X-KEY is read once: METHOD, URI, and the explicit IV. The target's key is relative and served with a .ts extension even though it is a key — the extension is never sniffed, because the URI comes from the key tag, not from a segment line.
• All other tags are ignored; the target's playlist is the media-playlist flavour, with no #EXT-X-STREAM-INF, so there is exactly one rendition and no variant-selection logic to write.

What is rejected rather than guessed

• A second #EXT-X-KEY: only a single playlist-level key is supported.
• AES-128 without an explicit IV: HLS would derive one from the media sequence number, which this client does not implement.
• Any method other than AES-128 or NONE — per-segment keys and SAMPLE-AES are errors, not approximations.

type Key struct { Method string; URI string; IV [16]byte }
type Segment struct { URI string; Duration float64 }
type Playlist struct { Key Key; Segments []Segment }
func Parse(src []byte) (Playlist, error)
func (k Key) Resolve(base *url.URL) (*url.URL, error)
func (s Segment) Resolve(base *url.URL) (*url.URL, error)

6 The downloader — internal/media

Everything after the playlist is one capability: retrieve, decrypt, and emit segments strictly in order, from a bounded window. Download never learns a file path — its output is the io.Writer it is handed, which is the whole boundary.

The media client pins HTTP/1.1

• The media host handles multiplexed HTTP/2 streams badly: one segment alone took about 15 seconds, and four concurrent requests never completed inside 120 seconds.
• The same four segments over HTTP/1.1, each on its own connection, finished in under 10 seconds. The client therefore forces HTTP/1.1 rather than letting ALPN choose.
• MaxIdleConnsPerHost equals the worker count (32): pinning it lower makes the pool evict and re-handshake connections continuously.
• There is no client-wide timeout; each attempt gets its own deadline inside Download.

The shape of a download

• The AES key is fetched once, up front, and must be exactly 16 bytes.
• One goroutine per segment. Before fetching, it must be admitted twice: the window requires the index to lie inside the sliding look-ahead, and the semaphore caps how many requests are in flight (default 32 — the measured knee: throughput scales about 2.6x from 4 to 32 connections, then flattens).
• An attempt is retried with exponential backoff (500ms doubling, capped at 15s, with jitter) after transients: 429, 5xx, and network errors. Other 4xx responses and decryption failures are final, because another attempt would fetch the same bytes.
• Decryption is AES-128-CBC with the playlist's one IV, and the PKCS#7 padding comes off. Leaving it on would end every segment at a length that is not a multiple of the 188-byte MPEG-TS packet size and corrupt the tail.
• A single emitter goroutine writes to dst in strictly ascending index order. It is the only writer, so the caller may tear down dst the moment Download returns.

The window, and why it is wider than the concurrency

The window is the bounded reorder buffer between the fetchers and the ordered writer: at most 4 x concurrency segments are resident by default. It is deliberately wider than the fetch limit, because latency against this host spans roughly 1.3 to 27 seconds; with the two equal, one slow segment at the write cursor blocks admission for everything behind it and the download runs at the speed of its slowest straggler — about half the throughput.

const DefaultConcurrency = 32
func NewClient() *http.Client
type Options struct {
    Concurrency    int           // 0 means 32; N greater than 0 pins exactly N in flight
    WindowSegments int           // 0 means 4 x Concurrency
    Referer        string        // sent on every request
    Client         *http.Client  // nil means http.DefaultClient; the seam for tests
    Retries        int           // per-segment retries after the first; 0 means 4
    AttemptTimeout time.Duration // per-attempt cap; 0 means 90s
    Progress       func(done, total int, bytes int64)
}
type Result struct { Bytes int64; Segments int }
func Download(ctx context.Context, pl hls.Playlist, playlistURL *url.URL, opts Options, dst io.Writer) (Result, error)
func DecryptSegment(key, iv, ciphertext []byte) ([]byte, error)

7 The final file — internal/output

The sink turns the ordered TS stream into the final playable file, and it does so while the download is still running. The .ts is always written — it is the source of truth and the fallback — and the same bytes are teed into a concurrent ffmpeg when one is on PATH.

The overlapped remux

• ffmpeg is started lazily on the first Write: a failure before the first byte leaves no child process and no temp file behind.
• The command is ffmpeg -hide_banner -loglevel error -y -i pipe:0 -c copy -bsf:a aac_adtstoasc -f mp4 base.mp4.part. -c copy means the picture and the sound are unchanged; the ADTS-to-ASC bitstream filter is what makes AAC valid inside MP4; and the format is named explicitly because the temp file's .part extension cannot tell ffmpeg the muxer.
• Because writes go straight into ffmpeg's stdin as they arrive, the remux overlaps the download; a successful run never re-reads the file.
• The final rename happens only after ffmpeg exits 0: .mp4.part becomes base.mp4 and the .ts is deleted. Nothing that looks complete is ever left behind.
• When ffmpeg is absent, the joined .ts is the product — most players accept it — and the run still succeeds.
• A dead ffmpeg surfaces as a write error, and media treats a write error as fatal rather than a retryable network fault.

Abort

On any failed run the sink kills ffmpeg if it was started, closes the pipe, and removes the partial .ts and .mp4.part, so the next run starts clean. If only the final remux fails, the script keeps the complete .ts instead — the download is not lost.

type Mode int
const ( Remuxed Mode = iota; KeptTS )
type Product struct { Path string; Mode Mode }
func TSPath(playlistURL *url.URL) string   // the .ts takes the playlist file's basename
func NewSink(ctx context.Context, tsPath, outBase string) (*Sink, error)
func (s *Sink) Write(p []byte) (int, error)
func (s *Sink) Finish() (Product, error)
func (s *Sink) Abort()
func (s *Sink) TSPath() string

8 The live lines — internal/progress

Two lines on stderr, both redrawn in place: a bar for the countable phase (segments) and a spinner for the phases that have no countable unit (the page read, the listing walk). The renderer draws; the caller decides when drawing is appropriate.

The console test

EnabledFor reports whether the writer is an interactive console. A piped or redirected run keeps byte-identical output: no carriage returns appear in a file. On Windows the check is GetConsoleMode, not os.ModeCharDevice — the latter also reports the NUL device, so goj … > NUL would be mistaken for a terminal and the bar would be drawn into nothing.

The bar

• Fixed shape: 30 cells, percent, done/total right-aligned to the total's width, cumulative bytes, and the whole-run average rate. Fixed widths stop the line from shivering as numbers change.
• Redraws are throttled to one per 100ms, except that the last segment always draws — the frame left on screen is the completed bar.
• No ETA and no instantaneous rate: the host's rate varies about twofold between runs, so both would be confidently wrong. The rate only appears after one full second, so the first frame can never print garbage.
• The bar is ASCII and counts bytes as columns on purpose: a code page reports an encoding, not font coverage, so nothing a program can check proves block glyphs render instead of boxes.

The spinner

A frame every 125ms (|/-\), with the elapsed seconds — the only honest signal a phase without a countable unit has that it has not hung. The page phase measures about a third of a second; a listing walk is one page read per page.

func New(w io.Writer, enabled bool) *Renderer
func EnabledFor(w io.Writer) bool
func (r *Renderer) Segment(done, total int, bytes int64)   // one more segment written
func (r *Renderer) StartSpinner(label string)              // the page / list phase
func (r *Renderer) StopSpinner()
func (r *Renderer) Clear()

9 The network and the limits

Two hosts with two very different behaviours, and one set of promises the tool makes about what it will and will not do.

What was measured, not assumed

Limits, stated plainly

• One playlist-level AES-128 key with an explicit IV. A playlist that omits the IV, uses a key per segment, or uses SAMPLE-AES is rejected with an error rather than guessed at.
• Extraction is a regex over the page source. A site that changes how it embeds the media URL needs a new entry in the rule tables.
• No login support: a URL that needs credentials will not work.
• A failed run removes its partial .ts and .mp4 and reports the error; if only the final remux fails, the complete .ts is kept.
• Download only what you have the right to download. You are responsible for how you use this.

The dependency budget

The module has exactly one dependency beyond the standard library: refraction-networking/utls, for the browser TLS fingerprints. ffmpeg is optional, found on PATH; without it the .ts is the product rather than an error.