Inside the Generator0%
Interactive field guideUpdated August 31, 2026

00 / The whole journey

From a sentence
to a signal.

Your prompt becomes a numerical condition; a generator uses that condition to reshape noise, masked tokens, or another starting representation; a decoder turns the result into media.

Here is the simple version.

A sentence is translated into machine-readable patterns. Those patterns guide a sampling process, which constructs an internal media representation before pixels or sound are decoded.

What is actually happening

  1. Text is split into tokens and encoded as contextual vectors.
  2. A generator starts from noise, masks, tokens, or an encoded reference.
  3. Attention or another conditioning mechanism connects the instruction to the evolving sample.
  4. Decoders and optional refinement stages produce the final media.
Prompt to generated media pipelineWords become tokens and embeddings, guide a latent generator, and decode into image, video, and audio.PROMPTa glass robotwalking throughviolet rainglassrobotrainGENERATORNOISE → STRUCTUREIMAGEVIDEOAUDIO
MYTH

Every generator follows one universal pipeline.

REALITY

Diffusion, flow, autoregressive, masked-token, and GAN systems take different routes—and production systems often combine them.

Technical layer Expand the formal version

A modern system may use several distinct spaces: text embeddings, image or audio latents, attention features, discrete tokens, and output pixels or waveform samples.

Key termstokenembeddingconditioninginference
Primary sourcesHigh-Resolution Image Synthesis with Latent Diffusion ModelsFlow Matching for Generative Modeling

01 / How models learn

How models learn

Training repeatedly compares a model’s prediction with a known target, then adjusts millions or billions of weights to reduce the error.

Here is the simple version.

Think of a vast practice session: example, guess, correction, tiny adjustment—repeated across many batches.

What is actually happening

  1. Media is collected, filtered, deduplicated, resized, cropped, captioned, or encoded.
  2. A batch passes through the network and produces a prediction.
  3. A loss measures error; backpropagation attributes it to parameters.
  4. An optimizer nudges weights, and the loop repeats.

Interactive / Training loop

Press the button to move one batch through a simplified update.

Model training loopExamples lead to predictions, loss, gradients, and updated weights.EXAMPLEPREDICTIONLOSSGRADIENTSWEIGHTSrepeat across batches
MYTH

Training builds a neat human-readable dictionary of concepts.

REALITY

The model learns distributed statistical structure and associations. Data scale does not guarantee quality, consent, or fair coverage.

Technical layer Expand the formal version

Training minimizes an objective L(θ) with gradient-based updates such as θ ← θ − η∇θL. Generation normally holds θ fixed while sampling a new output.

Key termsparameterlossbackpropagationtraining
Primary sourcesDenoising Diffusion Probabilistic ModelsLAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models

02 / Words become numbers

Words become numbers

A tokenizer splits text into pieces, and an encoder turns the sequence into context-dependent vectors.

Here is the simple version.

The words do not travel through the model as dictionary entries. They become numbered pieces whose representations shift with context and position.

What is actually happening

  1. Subword tokens receive integer IDs.
  2. An embedding layer maps IDs to vectors.
  3. Attention mixes context across the sequence.
  4. The final condition is usually a sequence of vectors, not one magical sentence number.

Interactive / Prompt pipeline

Teaching simulation—not real tokenization or embeddings.

IMAGE · SEED 42
aglassrobotwalkingthroughrain

tokens → contextual vectors → conditioned image generator

MYTH

Each prompt has one permanent coordinate.

REALITY

Representation depends on tokenizer, encoder, context, position, and model. Similar meanings can cluster without becoming identical.

Technical layer Expand the formal version

CLIP-style training raises similarity for matched image–text pairs and lowers it for mismatches; other systems use general language encoders or jointly trained multimodal encoders.

Key termstokenembeddingencodercross-attention
Primary sourcesLearning Transferable Visual Models From Natural Language SupervisionAttention Is All You Need

03 / Latent space

Latent space

Latent space is a compressed learned coordinate system where useful patterns can be represented more efficiently than raw media.

Here is the simple version.

It is like a map, a mixing desk, and a folded library—but every analogy breaks: real features are distributed, entangled, and model-specific.

What is actually happening

  1. Encoders map media into hidden representations.
  2. Nearby regions may correlate with related outputs.
  3. Interpolation blends latent states.
  4. A text embedding and image latent are not necessarily the same space.

Interactive / Latent atlas

Toy 2D projection—not a production model activation.

Toy projection of a latent spaceA sample moves among overlapping concept clusters in a two-dimensional teaching projection.glass / translucentmetallicnightlandscapeTeaching simulation — real latent spaces are high-dimensional.
Sample is between overlapping material and scene clusters.
MYTH

There is a universal ‘cat coordinate’ or ‘cinematic slider.’

REALITY

Some directions correlate with concepts, but dimensions are rarely clean, independent, or human-readable.

Technical layer Expand the formal version

Real latent spaces may contain hundreds or thousands of dimensions. Two-dimensional plots are projections that discard information and can distort neighborhood relationships.

Key termslatentencoderdecodertensor
Primary sourcesAuto-Encoding Variational BayesHigh-Resolution Image Synthesis with Latent Diffusion Models

04 / The VAE bottleneck

The VAE bottleneck

An autoencoder compresses media into a smaller learned representation so generation can happen with less computation.

Here is the simple version.

The encoder packs the perceptually important parts into a smaller grid; the decoder later reconstructs pixels or sound.

What is actually happening

  1. Raw media enters an encoder E.
  2. The compact latent z preserves useful structure.
  3. A generator edits or creates z.
  4. Decoder D reconstructs x̂, with some information loss.
Autoencoder compression bottleneckA pixel grid is compressed into a latent grid and expanded by a decoder.ENCODERDECODERless spatial data • learned features • cheaper generation
MYTH

Compression is lossless and only makes files smaller.

REALITY

Learned compression trades detail for efficiency; tiny text, faces, textures, or exact geometry can be lost.

Technical layer Expand the formal version

image x → encoder E → latent z → generator → decoder D → image x̂. A VAE balances reconstruction quality with regularization of a probabilistic latent distribution.

Key termsVAEautoencoderlatentdecoder
Primary sourcesAuto-Encoding Variational BayesHigh-Resolution Image Synthesis with Latent Diffusion Models

05 / Noise becomes structure

Noise becomes structure

Diffusion training learns a denoising direction; sampling applies that learned direction repeatedly, beginning from seeded noise.

Here is the simple version.

Imagine fogging examples by known amounts and training a restorer to estimate the fog. At generation time, begin with fog alone and iteratively organize it.

What is actually happening

  1. Training samples a clean example, time, and Gaussian noise.
  2. The network predicts noise, velocity, score, a clean sample, or a related target.
  3. Sampling uses a schedule and numerical update rule.
  4. A final latent is decoded into media.

Interactive / Sampling field

Conceptual geometry—not a working diffusion model.

Noise-to-data sampling and flow fieldNoise points form a geometric image while paths compare diffusion and flow.START: SIMPLE NOISESTRUCTURED SAMPLE
MYTH

Denoising retrieves the original training picture hidden inside the noise.

REALITY

Sampling constructs one possible output from learned statistical structure; memorization can still occur under some conditions.

Technical layer Expand the formal version

xₜ = √ᾱₜ x₀ + √(1−ᾱₜ)ε; a simple objective is E[‖ε−εθ(xₜ,t,c)‖²]. Parameterizations and solvers vary.

Key termsdiffusiondenoisernoise schedulesampler
Primary sourcesDenoising Diffusion Probabilistic ModelsClassifier-Free Diffusion Guidance

06 / Flow matching

Flow matching

Flow matching learns a time-dependent velocity field that transports samples from a simple source distribution toward data.

Here is the simple version.

Instead of learning to undo fog, learn the arrows of a current that carries points from noise toward structured samples.

What is actually happening

  1. Choose source and target samples with a probability path between them.
  2. Train a network to predict the path’s velocity.
  3. At generation time, an ODE solver follows the learned field.
  4. Rectified flow encourages straighter, easier-to-solve paths.
DIFFUSION VIEW

Reverse a noising process along a curved sequence of updates.

FLOW VIEW

Follow a learned velocity field through continuous time.

MYTH

Flow matching is just another name for diffusion.

REALITY

They are related continuous-time frameworks, but their objectives and sampling interpretations should not be collapsed.

Technical layer Expand the formal version

dx/dt = vθ(x,t,c), where x is the current representation, t continuous time, c the condition, and vθ the learned velocity field.

Key termsflow matchingrectified flowODEconditioning
Primary sourcesFlow Matching for Generative ModelingFlow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

07 / How text steers media

How text steers media

Attention lets features compare information, while cross-attention lets visual or audio features consult prompt tokens.

Here is the simple version.

Different parts of an emerging image can ask which words matter right now: color, material, subject, or spatial relation.

What is actually happening

  1. Self-attention relates positions within one sequence.
  2. Cross-attention forms queries from media features and keys/values from text.
  3. Weights modulate which token information is injected.
  4. Spatial-language binding remains imperfect.

Interactive / Cross-attention

Select a token to emphasize conceptual connections.

Cross-attention between text and visual featuresPrompt tokens connect with varying strength to regions of an emerging image.aredglassrobotbeneathamoon
MYTH

Attention maps reveal the model’s full reasoning.

REALITY

They are informative signals, not complete causal explanations of a network’s behavior.

Technical layer Expand the formal version

Earlier systems often used U-Net denoisers; DiTs treat latent patches as tokens. Transformer describes an architecture, not a generation objective.

Key termsself-attentioncross-attentionU-NetDiT
Primary sourcesAttention Is All You NeedScalable Diffusion Models with Transformers

08 / Seeds, steps, guidance

Seeds, steps, guidance

Seeds set the pseudorandom start, samplers choose updates, and guidance trades diversity for stronger conditioning.

Here is the simple version.

Controls influence the route through sampling; none is a guaranteed quality dial.

What is actually happening

  1. A seed initializes noise deterministically within a software path.
  2. Steps provide more numerical evaluations, with diminishing returns.
  3. A solver decides where and how to update.
  4. Guidance amplifies the difference between conditioned and less-conditioned predictions.

Interactive / Guidance

Amplifying prompt direction often narrows diversity.

SCALE 6.0
conditioned direction
DIVERSITY
MYTH

More steps or guidance always means a better image.

REALITY

High guidance can cause artifacts and lower diversity; efficient solvers may need fewer steps.

Technical layer Expand the formal version

guided = unconditioned + scale × (conditioned − unconditioned). Negative prompts alter a condition; they are not guaranteed forbidden-object lists.

Key termsseedsamplerclassifier-free guidanceinference
Primary sourcesClassifier-Free Diffusion GuidanceConsistency Models

09 / Editing and control

Editing and control

Editing changes selected information while trying to preserve identity, structure, appearance, or context.

Here is the simple version.

Every edit is a negotiation: give the model freedom to change, while anchoring what must remain.

What is actually happening

  1. Image-to-image adds noise to an encoded input before resampling.
  2. Inpainting regenerates masked regions; outpainting extends boundaries.
  3. Structural controls supply edges, depth, pose, or segmentation.
  4. Adapters and fine-tuning alter how a model responds.

Interactive / Preservation triangle

Available appearance preservation: 45%

The toy constraint makes the trade-off visible; real systems behave less neatly.

MYTH

Prompt, structure, and appearance preservation can all be maximized at once.

REALITY

Because features are entangled, stronger change can disturb identity, geometry, text, or background.

Technical layer Expand the formal version

Inversion estimates a latent or noise trajectory for an existing image. ControlNet-like branches and reference adapters add conditioned spatial or appearance features.

Key termsinpaintinginversionControlNetadapter
Primary sourcesAdding Conditional Control to Text-to-Image Diffusion ModelsInstructPix2Pix: Learning to Follow Image Editing Instructions

10 / Adding time

Adding time

Video generation adds time, requiring convincing frames plus consistent identity, motion, lighting, and camera behavior.

Here is the simple version.

One good frame is not enough; the story must remain stable as the sequence unfolds.

What is actually happening

  1. Compressed video can become spacetime patches.
  2. Transformers exchange information across space and time.
  3. Base clips may receive spatial and temporal super-resolution.
  4. Conditioning may include text, keyframes, motion, or camera paths.

Interactive / Video time

Scrub through five conceptual frames.

FRAME 3/5
Video transformed into spacetime patchesVideo frames form a volume, divide into patches, enter a transformer, and reconstruct.
Subject identity is comparatively stable.
MYTH

A video model simply renders independent pictures in a loop.

REALITY

Many systems model multiple frames or spacetime tokens jointly, though architectures vary.

Technical layer Expand the formal version

Video adds many more tokens and dimensions. Errors compound across time, so long-range consistency is substantially harder than local frame quality.

Key termstemporal consistencyframe interpolationsuper-resolutiontransformer
Primary sourcesVideo Diffusion ModelsVideo Generation Models as World Simulators

11 / Generating sound

Generating sound

Audio generators model waveforms, spectrograms, codec tokens, or continuous latents, then decode them into sound.

Here is the simple version.

Long-range tokens can organize words, rhythm, or melody while lower-level codes carry timbre and acoustic detail.

What is actually happening

  1. Text may predict an acoustic representation for a vocoder.
  2. Autoregressive models can predict discrete codec tokens.
  3. Diffusion or flow can generate noisy spectrogram or audio latents.
  4. Codec decoders or vocoders reconstruct waveforms.

Interactive / Audio tokens

No sound is generated or autoplayed.

Audio codec token generation pipelineAudio is encoded into semantic and acoustic tokens, modeled, then decoded.ENCODER
MYTH

All sound generators directly draw the final waveform.

REALITY

Most use compressed, hierarchical, or time-frequency representations because raw audio is an extremely long sequence.

Technical layer Expand the formal version

Speech adds pronunciation, prosody, language transfer, identity, consent, and impersonation risk; music adds long-range form, rhythm, harmony, lyrics, and rights questions.

Key termsneural codecquantizationVQ-VAE / VQGANautoregressive
Primary sourcesSoundStream: An End-to-End Neural Audio CodecAudioLM: a Language Modeling Approach to Audio Generation

12 / Model families

Model families

Generative families differ in what they start from, what they learn, and how they sample.

Here is the simple version.

VAE, GAN, token, diffusion, and flow approaches are different tools; hybrids combine their strengths and complexity.

What is actually happening

  1. VAEs learn probabilistic compression.
  2. GANs train a generator against a discriminator.
  3. Autoregressive and masked systems predict tokens.
  4. Diffusion and flow learn iterative or continuous transport.
Map of generative model familiesSeveral frameworks connect through shared representations and architectures.HYBRIDreal systems combineVAEGANAUTOREGRESSIVEMASKED TOKENDIFFUSIONFLOW
FamilyStarts fromLearnsGenerates byStrengthsWeaknesses
VAEencoded dataprobabilistic latentsampling + decodingsmooth compressionreconstruction loss
GANrandom vectorgenerator–discriminator gameone forward passfast, sharpinstability, mode collapse
Autoregressiveearlier tokensnext-token distributionsequential predictioncoherent token modelingslow for long sequences
Masked-tokenpartially masked tokensmissing-token predictionparallel fillingfewer roundstokenizer-dependent
Diffusionnoisy samplereverse directioniterative denoisingfidelity, conditioningmany evaluations
Flow matchingsource samples + pathsvelocity fieldODE integrationefficient pathssolver design matters
Hybridmultiple representationscombined objectivesmultiple stagescombines strengthsadded complexity
MYTH

Transformer, diffusion, and autoregressive are mutually exclusive labels.

REALITY

Transformer is an architecture; diffusion, flow, autoregression, and masked prediction are training or generation frameworks.

Technical layer Expand the formal version

A Transformer can power a diffusion denoiser, a flow velocity field, or a next-token predictor. Model taxonomy has multiple independent axes.

Key termsGANVAEdiffusiontransformer
Primary sourcesGenerative Adversarial NetworksMaskGIT: Masked Generative Image Transformer

13 / Where models fail

Where models fail

Generators optimize statistical plausibility, not guaranteed truth, counting, physics, identity, or exact symbolic correctness.

Here is the simple version.

A fluent-looking result can still be structurally wrong, biased, misleading, or fabricated.

What is actually happening

  1. Training distributions are uneven.
  2. Compressed representations discard detail.
  3. Concept binding and spatial relations can fail.
  4. Long sequences multiply consistency demands.
PLAUSIBLE≠TRUE

Fluency and visual polish are not evidence.

MYTH

If an output looks convincing, its content is reliable.

REALITY

Plausible is not the same as true. Inspect text, anatomy, identity, timing, causality, and factual claims.

Technical layer Expand the formal version

Metrics such as FID compare distributions through learned image features; they capture only part of quality and can hide subgroup or semantic failures.

Key termsFIDtemporal consistencyconditioninglatent
Primary sourcesGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash EquilibriumExtracting Training Data from Diffusion Models

14 / Rights and provenance

Rights and provenance

Capability does not settle questions of consent, licensing, privacy, impersonation, bias, or authenticity.

Here is the simple version.

Responsible use needs both technical safeguards and social rules: document data, limit misuse, and preserve trustworthy origin records.

What is actually happening

  1. Training-data documentation supports scrutiny.
  2. Memorization and extraction risks require testing.
  3. Watermarks and signed manifests can record declared history.
  4. Moderation and logging choices create their own privacy trade-offs.
Signed media provenance chainCreation and editing events form a signed chain leading to verification.CREATEsigned event 1EDITsigned event 2EXPORTsigned event 3VERIFYsigned event 4Provenance can report origin and edits.It does not prove that the depicted event is true.
MYTH

A provenance badge proves that the depicted event happened.

REALITY

C2PA-style provenance can report declared origin and edits; it cannot by itself establish that depicted content is true.

Technical layer Expand the formal version

Generative models usually produce new samples from learned structure, but ‘usually’ is not ‘never memorize.’ Extraction has been demonstrated under some conditions.

Key termsprovenancefine-tuningparameterinference
Primary sourcesExtracting Training Data from Diffusion ModelsC2PA Technical Specification 2.4

15 / Glossary

Terms without the fog.

Search the vocabulary that appears throughout the guide.

parameter

A learned numerical value inside a model.

Think of it as
One adjustable screw among billions.
Do not confuse with
A user setting.

weight

A parameter that scales how strongly one signal affects another.

Think of it as
A mixing-desk fader.
Do not confuse with
Physical weight.

tensor

A multidimensional array of numbers.

Think of it as
A spreadsheet extended into more dimensions.

token

A discrete unit processed by a model, such as a subword or codec symbol.

Think of it as
A labeled tile in a sequence.
Do not confuse with
Always a full word.

embedding

A learned vector representation.

Think of it as
Coordinates on a model-specific map.
Do not confuse with
A universal concept coordinate.

latent

A hidden learned representation.

Think of it as
A compact working sketch.
Do not confuse with
The text embedding.

encoder

A network that maps input into a representation.

Think of it as
A careful packing machine.

decoder

A network that maps a representation back to usable media.

Think of it as
An unpacking and reconstruction machine.

autoencoder

An encoder–decoder trained to reconstruct its input.

Think of it as
Compress, then rebuild.

VAE

A variational autoencoder with a regularized probabilistic latent space.

Think of it as
A compressor that learns smooth regions for sampling.
Do not confuse with
A plain deterministic codec.

VQ-VAE / VQGAN

An autoencoder that uses a discrete learned codebook.

Think of it as
Describing media with learned tiles.

quantization

Mapping continuous values to discrete codes.

Think of it as
Rounding into a learned vocabulary.

neural codec

A learned encoder and decoder for compressed audio or media.

Think of it as
A model-trained zip format.

diffusion

A framework that learns to reverse a noising process.

Think of it as
Learning to clear measured fog.

score

A field related to the gradient of log probability.

Think of it as
Arrows pointing toward denser regions.

denoiser

A network that predicts a reverse sampling direction.

Think of it as
A trained restoration guide.

noise schedule

A rule describing noise levels across time.

Think of it as
The sampling route’s timetable.

sampler

The numerical procedure that updates a generative sample.

Think of it as
The driver following learned directions.

ODE

An ordinary differential equation describing continuous deterministic change.

Think of it as
A rule for following a current.

SDE

A stochastic differential equation that includes random change.

Think of it as
A current with controlled turbulence.

flow matching

Training a velocity field between source and data distributions.

Think of it as
Learning the arrows of a transport current.

rectified flow

A flow formulation that encourages straighter paths.

Think of it as
Straightening a winding route.

transformer

An architecture built around attention and token processing.

Think of it as
A room where every token can consult others.
Do not confuse with
A generation objective.

self-attention

Attention within one sequence.

Think of it as
Words consulting neighboring words.

cross-attention

Attention from one representation to another.

Think of it as
Image features consulting prompt tokens.

U-Net

A multiscale encoder–decoder architecture with skip connections.

Think of it as
Zoom out, process, and restore detail.

DiT

A Diffusion Transformer that processes latent patches as tokens.

Think of it as
A Transformer acting as the denoiser.

conditioning

Information that steers generation.

Think of it as
A set of directions for the sampler.

classifier-free guidance

Amplifying the gap between conditioned and less-conditioned predictions.

Think of it as
Turning up prompt influence.

seed

A number initializing a pseudorandom process.

Think of it as
A repeatable starting shuffle.
Do not confuse with
A compressed image.

inference

Using fixed learned weights to produce a prediction or sample.

Think of it as
Taking the trained machine for a run.
Do not confuse with
Training.

training

Adjusting parameters to reduce a loss over examples.

Think of it as
The long practice phase.

loss

A numerical measure of prediction error.

Think of it as
A score telling the model how far off it was.

backpropagation

Computing how parameters contributed to a loss.

Think of it as
Tracing error backward through the machinery.

fine-tuning

Further training an existing model on a narrower objective or dataset.

Think of it as
Specialized follow-up practice.

adapter

A small trainable module added to a model.

Think of it as
A detachable specialization layer.

inpainting

Regenerating a masked region using surrounding context.

Think of it as
Repairing one part of a canvas.

inversion

Estimating a latent or sampling path for an existing output.

Think of it as
Finding a route back into editable model space.

ControlNet

A conditioned branch that adds spatial control to diffusion models.

Think of it as
A tracing guide for structure.

frame interpolation

Creating intermediate frames between existing frames.

Think of it as
Filling steps between snapshots.

super-resolution

Increasing resolution with a learned model.

Think of it as
Reconstructing a more detailed larger version.

temporal consistency

Stability of identity and appearance across time.

Think of it as
Keeping the same actor and props between frames.

FID

A distribution-level image metric based on learned features.

Think of it as
Comparing two crowds rather than single images.
Do not confuse with
A complete quality score.

provenance

Records describing media origin and modification history.

Think of it as
A signed chain of custody.
Do not confuse with
Proof that depicted events are true.

16 / Primary sources

Follow the evidence.

Research papers and official specifications used for the lessons above.

Latent representations · 2013

Auto-Encoding Variational Bayes

D. P. Kingma and M. Welling

VAEs, probabilistic latent variables, and reparameterization.

https://arxiv.org/abs/1312.6114
Transformers · 2017

Attention Is All You Need

A. Vaswani et al.

Transformer architecture and attention-based sequence modeling.

https://arxiv.org/abs/1706.03762
Video · 2022

Video Diffusion Models

J. Ho et al.

Extending image diffusion architectures to coherent video.

https://arxiv.org/abs/2204.03458
Fast generation · 2023

Consistency Models

Y. Song et al.

One- and few-step alternatives or distillations for diffusion.

https://arxiv.org/abs/2303.01469
Provenance · Version 2.4

C2PA Technical Specification 2.4

Coalition for Content Provenance and Authenticity

Signed manifests and interoperable media provenance.

https://spec.c2pa.org/specifications/specifications/2.4/specs/C2PA_Specification.html