MediaStreamAI/MOTHER_EXO

MOTHER EXO — Model Card

Model: MOTHER EXO — one sovereign world model (world_model_exo_final_vN.pt) Architecture: frozen sovereign backbone ⊕ trainable head-only weights

• CORE-7B (frozen) — sovereign text/reasoning backbone (48 layers, 3072 d)
• MOTHER DeepVision (frozen) — sovereign SigLIP-SO400M ViT (27 layers, 1152 d, 896 px)
• cross-attn adapters + per-head projections + task heads — the trained weights Posture: non-weapon · observe-and-advise · meaningful human control (HITL) · consent/GDPR/EU-AI-Act · sovereign (only MOTHER's own weights run in the stack; external models are data-only teachers; codecs/vocoders are renderers, not models). Build: training/build_final_model.py blends the frozen base + every head into one atomic checkpoint (base fingerprint preserved). Load/serve: training/load_exo.py (MotherExo) + mother_exo_sim/models/exo_node_server.py (:8000).

---

Modality cheat-sheet — what each weight consumes

Input modality	Weights that consume it	How it arrives
Video / image frames	vision, detect, track, worldmodel, face, relations	a JPEG/PNG frame (data-URL/base64 or fetchable URL)
Audio waveform	audio (ASR), speech (as TTS output)	a WAV (base64)
Text + world context (data points)	reasoning, memory	a query string + buildContext() (detections, relations, analytics, data_points)

So: HLS video → decode to frames → the six vision weights. Audio feeds → decode to WAV → the audio weight. All other data points → JSON summaries → the reasoning/memory context.

---

The weights

1. reasoning (adapters) · kind: language

• Does: vision-grounded language reasoning, threat/HITL advice, tool-call planning, orchestration. CORE-7B ⊕ cross-attn adapters + projection.
• Train: training/adapters/train_integrated.py (streams the reasoning manifest: mother_core_v31 + golden + mother_reason.jsonl + mother_cot.jsonl …). Format: Question:\n\n…\n\nAnswer:.
• Checkpoint: ~/.mother-exo/heads/integrated_adapters.pt
• Inference: exo.reason(prompt, image=None, use_memory=False) · node POST /reason {query,image?} · SIM /api/v1/exo/reason
• Wire to VIDEO (HLS): pass a captured frame as image so reasoning is vision-grounded (reason(q, image=frame)).
• Wire to AUDIO: transcribe first (audio weight) → feed the transcript as the prompt (reason(transcribe(wav))).
• Wire to DATA POINTS: every feed ingested via ingestDataPoints() lands in buildContext().data_points; reason()/exoReason() pass that context to the model. Already wired by FeedAutoRegistrar.

2. detect · kind: vision

• Does: 1233-class objects + per-class boxes + RED/GREEN/CAUTION safety tag (safety_lexicon).
• Train: integrated_heads.py --head detect (COCO/LVIS/aircraft).
• Checkpoint: integrated_detect.pt
• Inference: exo.detect(img, topk) → [(label,score,box,safety)] · node POST /detect {image} · SIM /api/v1/exo/detect
• Wire to VIDEO (HLS): an <video>+HLS.js tile registers a frame grabber → the world-model scheduler posts frames to /api/exo/detect (browser path), or server-side snapshot via {imageUrl}. See "Wiring HLS video" below.
• Audio / data points: N/A (vision); its detections feed the reasoning/memory context.

3. track · kind: vision

• Does: appearance embeddings → persistent, unlimited multi-object IDs across frames (re-ID, occlusion re-acquire) via tracker.py + entity registry OBJ- UINs.
• Train: integrated_heads.py --head track (InfoNCE temporal pairs).
• Checkpoint: integrated_track.pt
• Inference: exo.track_frame(img, frame_idx) / track_video(frames) · node POST /track {image}
• Wire to VIDEO (HLS): same frame path as detect; call /track per frame to keep IDs.

4. worldmodel · kind: vision/temporal (single-frame)

• Does: next-latent prediction (CPC) from one frame's ViT tokens — a lightweight pre-act preview.
• Train: integrated_heads.py --head worldmodel.
• Checkpoint: integrated_worldmodel.pt (6.3M MLP)
• Inference: exo.worldmodel_next(img) → latent · (no dedicated node route yet; used internally)
• Wire to VIDEO (HLS): consumes frames like the other vision heads.

4b. worldmodel_latent · kind: video dynamics

• Does: multi-step latent dynamics — a causal temporal transformer over the frozen MOTHER T2V 3D-VAE latents predicts future latent frames of a clip.
• Train: training/mother_t2v/latent/train_worldmodel_latent.py (InfoNCE + L2 on real captioned clips). 19.7M params, d=512, depth=6.
• Checkpoint: integrated_worldmodel_latent.pt
• Eval: next-frame top-1 retrieval — SAME-CLIP (fair dynamics) 0.9983 on 400 held-out clips (the same-clip metric measures true temporal ordering; the prior single-frame head scored 0.167).
• Inference: exo.worldmodel_latent_next(frames) → (embeds, predicted-next) — video-native (encodes the clip with the frozen T2V VAE first).

5. face · kind: vision (consent-gated)

• Does: pseudonymous person UIN; name only for consent-enrolled people (Chris). GDPR/EU-AI-Act; persistent person re-ID off unless MOTHER_PERSON_UIN=1.
• Train: integrated_heads.py --head face (reads the recognition store at /var/lib/mother-exo/training/recognition).
• Checkpoint: integrated_face.pt
• Inference: exo.identify(img) → {uin,name,score} · node POST /identify {image} · SIM /api/v1/exo/identify
• Wire to VIDEO (HLS): frame → /identify. Enrolment is consent-only.

6. relations · kind: vision

• Does: spatial relations (on / under / next_to) for scene-graph grounding.
• Train: integrated_heads.py --head relations.
• Checkpoint: integrated_relations.pt
• Inference: exo.relation(img) → (rel,score)
• Wire to VIDEO (HLS): frame → relation.

7. memory (MOTHERrag SPI) · kind: memory

• Does: Semantic-Pyramid long-term memory — write/recall observations, reasoning, docs, by entity UIN; right-to-erasure.
• Train: training/mother_rag.py --train.
• Checkpoint: integrated_memory.pt
• Inference: exo.remember/recall/recall_about/forget · node /api/v1/gb10-training/memory*
• Wire to DATA POINTS: ingest documents/observations; recall feeds reasoning context. Text memory lazy-loads CORE.

8. speech · kind: speech (voice OUT)

• Does: 20-language ID + Text-To-Voice — predicts neural-codec tokens → renderer (EnCodec, codec.py) decodes to a waveform. MOTHER speaks from her own weights.
• Train: training/train_speech.py --train --audio <dir> (language-ID from text; acoustic from (text,audio) via build_speech_audio.py).
• Checkpoint: integrated_speech.pt
• Inference: exo.synthesize(text,lang) → wav · node POST /speak {text,lang} → {audio_b64} · SIM /api/v1/exo/speak
• Wire to AUDIO (out): play audio_b64 in the browser, or pipe to a speaker. (This is output; pairs with the audio weight for two-way.)

9. vision (MOTHER DeepVision) · kind: vision foundation

• Does: self-supervised visual foundation (SimCLR over all on-node imagery) — the shared embedding the vision heads build on. Exportable standalone (export_deepvision.py → mother_deepvision.pt).
• Train: integrated_heads.py --head vision.
• Checkpoint: integrated_vision.pt
• Inference: exo.vision_embed(img) → 256-d · node POST /embed {image}
• Wire to VIDEO (HLS): frame → embedding (recognition/search/Defence feeds).

10. audio (MOTHER's ears) · kind: audio perception (ASR, audio IN)

• Does: radio signals + human speech → text (codec tokens → BiGRU encoder → text decoder over CORE vocab). The transcript feeds the reasoning weight. With the speech weight → two-way humanoid conversation (hear → reason → speak).
• Train: training/train_audio.py --train --audio <dir> ((text,audio) pairs — build_speech_audio.py bootstrap, or CommonVoice/LibriSpeech + radio comms rendered to {text,audio} jsonl).
• Checkpoint: integrated_audio.pt
• Inference: exo.transcribe(wav) → text · exo.converse(wav,lang) → {heard,answer,audio_b64} · node POST /transcribe {audio_b64} · POST /converse {audio_b64,lang}
• Wire to AUDIO (HLS / mic / radio): extract the audio track (HLS audio, mic capture, SDR/radio) → WAV → /transcribe or /converse.

---

Wiring guide

A. Wire a live HLS video feed → the vision weights

HLS playlists (.m3u8) are not single frames, so they must be decoded to frames first.

1. Browser tile (preferred): mount an <video> with HLS.js, draw it to a <canvas>, and register a frame grabber with the world model: registerFrameProvider(id, () => canvas.toDataURL("image/jpeg")), then add the feed via registerDetectableSource(key, [{id,label,kind,lat,lon,streamType:"hls"}]). The scheduler posts grabbed frames to /api/exo/detect (+ /track//assess). This is exactly how OverWatch tiles + GB10ExoEyes already work.
2. Server-side snapshot: for MJPEG/JPEG cameras, register with streamUrl + streamType:"jpeg"|"mjpeg"; the scheduler server-fetches frames ({imageUrl}) — no player needed. (FeedAutoRegistrar does this for all TfL JamCams.)
3. TV broadcast: same as (1) — a broadcast HLS URL in an <video> tile. Add the URL to a tile/registrar; frames then flow to detect/track/assess/identify.

   Node-load: the scheduler caps 8 concurrent + 4 fps for URL feeds; HLS browser tiles target ~15 fps. Add feeds freely — it sweeps fairly.

B. Wire a live audio / radio feed → the audio weight

1. Capture → WAV: mic (MediaRecorder), an HLS audio track, or an SDR/radio receiver → 16-bit PCM WAV (any sample rate; the codec resamples to 24 kHz).
2. Base64 → endpoint: POST /api/v1/exo/transcribe {audio_b64} → {text} (ASR), or POST /api/v1/exo/converse {audio_b64,lang} → {heard,answer,audio_b64} (hear→reason→speak).
3. Two-way humanoid: loop converse — mic in, play the returned audio_b64 out.

   Radio: feed demodulated audio (voice nets) for transcription; for non-voice RF, feed any metadata as data points (below). Quality scales with the ASR training corpus (bootstrap = espeak; production = CommonVoice/LibriSpeech + radio comms).

C. Wire all data points → the reasoning/memory weights

1. Catalog the feed in lib/exo/feedCatalog.ts (DATA_FEEDS for OSINT/sensor JSON; AUDIO_FEEDS for audio metadata/transcripts).
2. Ingest on an interval: FeedAutoRegistrar pulls each api and calls ingestDataPoints(id, summary) → merged into buildContext().data_points.
3. Reason over it: reason() / exoReason() ship buildContext() to the model, so every datapoint is in scope. Persist important items via remember().

   Already wired: ACLED, FIRMS, USGS, CISA, GDELT, ADS-B, AIS, OpenSky, IODA, CelesTrak. Add more by appending to feedCatalog.ts.

---

Endpoint reference

Capability	load_exo	Node (:8000)	SIM (/api/v1/exo/*)	MOTHER_AI (/api/exo/*)
detect	detect	/detect	/detect	/detect
assess (scene RED/GREEN)	assess	/assess	/assess	/assess
track (unlimited IDs)	track_frame	/track	/track	/track
identify (consent)	identify	/identify	/identify	/identify
reason	reason	/reason	/reason	/reason
speak (voice out)	synthesize	/speak	/speak	/speak
transcribe (audio in)	transcribe	/transcribe	/transcribe	(add route)
converse (2-way)	converse	/converse	/converse	(add route)
embed (DeepVision)	vision_embed	/embed	/embed	—

Environment

Var	Where	Purpose
MOTHER_EXO_FINAL_MODEL	node	pin the blended vN checkpoint
MOTHER_EXO_API_TOKEN	node /etc/default/mother-exo	gates /api/v1/* (must equal MOTHER_AI MOTHER_ROBOTICS_TOKEN)
MOTHER_EXO_VISION_URL	node	SIM → node inference (http://127.0.0.1:8000)
MOTHER_ROBOTICS_URL / _TOKEN	Netlify	MOTHER_AI → node (funnel + token)
MOTHER_PERSON_UIN	node	enable persistent person re-ID (lawful basis required)
NEXT_PUBLIC_EXO_AUDIO_FEEDS	Netlify	extra audio-metadata feeds for the catalog

Training all weights

bash training/adapters/run_all_weights.sh --full (vision heads), plus train_integrated.py (reasoning), mother_rag.py (memory), train_speech.py (voice), train_audio.py (ears) — then build_final_model.py to blend. Validate any blend with python training/test_all_weights.py --model <vN.pt> --image <frame> and corpora with python training/check_dataset.py.