Independent guide · Applies to the July 2026 LingBot-Video release · Last verified: July 12, 2026

LingBot-Video prompt guide: structured JSON captions, not plain text

The single most important thing to know about LingBot-Video: its DiT inference consumes structured JSON captions, not casual prompts. The official workflow runs every prompt through a two-stage rewriter first. Here is the complete pipeline, the commands for each mode, and the gotchas.

The pipeline at a glance

your plain prompt (+ first frame, TI2V only)
        │
        ▼
  step 1 · EXPAND   base VLM (Qwen3.6-27B)          → detailed caption
        │
        ▼
  step 2 · MAP      same VLM + rewriter LoRA        → structured JSON caption
        │
        ▼
  (optional) auto-negative                          → negative.json
        │
        ▼
  scripts/inference.py --prompt_json prompt.json    → video

Both rewriter stages share the same base VLM; step 2 attaches the official LoRA adapter. For TI2V (image-to-video), the first-frame image is fed to both stages — and must be the same image you later pass to DiT inference.

Power-user tip (from the official docs): step 1 EXPAND does not depend on the LoRA, so you can swap in a stronger VLM for the expand stage to get a richer intermediate caption — then run step 2 MAP with the official base + LoRA as usual.

What the structured JSON looks like

{
  "caption": {
    "...": "structured JSON caption"
  },
  "duration": 5
}

The repo ships complete real examples under assets/cases/<mode>/example_*/prompt.json for t2i, t2v, and ti2v — read a few before writing anything by hand.

Timestamp gotcha: the caption carries per-action timestamps like [0.0s - 5.0s] under prominent_elements[].actions[]. Your --duration must match the total span those timestamps cover. If the latest action reaches 5.0s, set --duration 5. When prompt.json contains duration, the runner derives num_frames from duration × fps automatically.

Rewriter commands for each mode

Text-to-video (T2V)

python rewriter/inference.py \
  --backend transformers \
  --mode t2v \
  --prompt "<plain_user_prompt>" \
  --duration 5 \
  --output prompt.json

Image-to-video (TI2V)

Use the same first frame here and later in DiT inference:

python rewriter/inference.py \
  --backend transformers \
  --mode ti2v \
  --prompt "<plain_user_prompt>" \
  --first-frame "<first_frame.png>" \
  --duration 5 \
  --output prompt.json

Text-to-image (T2I)

python rewriter/inference.py \
  --backend transformers \
  --mode t2i \
  --prompt "<plain_user_prompt>" \
  --output prompt.json

Set REWRITER_BASE_MODEL and REWRITER_ADAPTER environment variables to the downloaded weights first. The bundled transformers backend needs no extra install beyond requirements.txt.

Auto-negative: optional, delete-only

LingBot-Video ships default negative prompts (one for video, one for image). Auto-negative reads your structured caption and removes default-negative terms that conflict with what you actually asked for. It never invents new negative terms.

python rewriter/auto_negative.py \
  --backend transformers \
  --mode t2v \
  --caption prompt.json \
  --output negative.json

Pass --negative_prompt_json negative.json to inference, or omit it entirely to use the built-in default. Official guidance: the default is enough in most cases.

Deploying the rewriter in production

The bundled transformers backend is a single-process reference implementation — one request at a time. For production, the official docs recommend serving the VLM behind an OpenAI-compatible API (vLLM or SGLang) while preserving the two-stage semantics:

• Step 1 must hit the base VLM without the rewriter LoRA.
• Step 2 must hit the same base VLM with the LoRA enabled.
• Implement as two endpoints, or one server that selects the adapter per request.

Frequently asked questions

Next: install and run your first generation → · Back to the LingBot-Video model guide