# Assets and formats Source: https://zafnok.github.io/shuyi-engine/learn/book/assets-and-formats/ A game is made of more than code. It has words, keys, colors, maps, pictures and dialogue, and you want to change those without touching the code, and perhaps let players change them too. Each of these is an **asset**, and this chapter is about how a game gets from a file to a value it can use. Change one number in a file, and the game follows, with no code changed: In `files/garden.ron`: ```diff - plots: 6, + plots: 10, ``` ![The garden has six plots in its row.](https://zafnok.github.io/shuyi-engine/tutorial-tries/book-plan-plots/before.png) ![The garden has ten plots in its row, still centred. No code changed.](https://zafnok.github.io/shuyi-engine/tutorial-tries/book-plan-plots/after.png) Everything a game reads from files comes through three parts: - a **kind** of asset is a plain type, what you want from the file: a palette, an image, the garden's plan; - a **format** reads one kind from a file's bytes, and says every problem with its file and line; - the **registry**, `Formats`, holds the formats, and finds an asset by its **id** in the game's **bundle**, all its files together. The code that asks for an asset never sees which format read it, so a faster or smaller format can come later without touching that code. Standard formats (PNG, WAV, Ogg Vorbis) always load, and the text formats (RON files, `.map` grids, `.dlg` dialogue) stay the source of what is in a game. ## A kind of the game's own The garden's plan is a kind of the game's own, plain data with a name for messages: ```rust /// The garden's plan: a kind of asset of the game's own, plain data. #[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] #[serde(deny_unknown_fields)] pub struct Plan { /// How many plots the garden has. pub plots: u8, /// How many seeds the player starts with. pub seeds: u32, /// How often a seed grows, in percent. pub grow_percent: u8, } impl AssetKind for Plan { const NAME: &'static str = "garden plan"; } ``` > **Rust:** `u8` and `u32` are whole numbers that are never negative, 8 and 32 bits > long, so a `u8` goes up to 255. `Deserialize` comes from a crate called `serde`, and > deriving it writes the code that reads this type from a file. > `#[serde(deny_unknown_fields)]` says a field this type doesn't have is an error > rather than being ignored, which catches a typo in the file. `impl AssetKind for Plan` > keeps the one promise of that trait, a `const`, a fixed value: its name in messages. in a RON file of the game's: ```ron // The garden's plan: how many plots it has, how many seeds the player // starts with, and how often a seed grows, in percent. ( plots: 6, seeds: 9, grow_percent: 75, ) ``` ## Reading it through the registry The game registers a RON format for its kind beside the engine's, and asks the registry for the plan by its id, the file's path with or without its extension: ```rust /// The formats the game reads its files with: the short path's, and a /// RON file for its plan. pub fn formats() -> Formats { let mut formats = BasicData::::formats(); // The first format of its kind: never refused. let _ = formats.register(Ron::::new()); formats } /// Reads the game's own part, its plan, from `garden.ron` (the id needs /// no extension: the registry finds the file a format of the kind reads), /// then checks what the format can't. pub fn read_plan(bundle: &Bundle) -> Result> { let plan = formats().load::(bundle, "garden")?; if plan.grow_percent > 100 { let file = bundle.display_path("garden.ron"); return Err(vec![ContentError::new(file, "grow_percent is over 100")]); } Ok(plan) } ``` > **Rust:** `Ron::::new()` makes the format that reads a `Plan` from RON. `let mut` > gives a name to a value that may change, and `register` changes it. > > `Result>` is either a `Plan`, or the list of what's wrong. > The `?` after `.load::(bundle, "garden")` says: if this is an error, hand it > back now, and otherwise give me the value inside. `Err(vec![...])` is the same kind of > error, made by hand, and `Ok(plan)` is the good answer. > > `&Bundle` means this function borrows the bundle, only to look. `read_plan` is the function `BasicGame::with_own` takes in [chapter 1](https://zafnok.github.io/shuyi-engine/learn/book/quick-start/index.md#what-it-is-made-of): it reads the game's own part when the game starts. A check the format can't make (a percent over 100) comes after loading, and names the file as the bundle shows it. `cargo shuyi check` loads every file the game reads and gives every error at once, each with its file, line and problem code. `cargo shuyi import ` reads one file as every kind whose formats know it, and says what it read. ## When a file is wrong A broken file should never crash a game, or be half read. A format finds each problem and says where it is, and the tools say how to fix it. These are real messages from Moon Garden's `garden.ron`. A **problem code**, like `SH2001`, has a page of its own that `cargo shuyi explain SH2001` prints. The wrong type, `grow_percent: "lots"`: ```text check: 1 error(s) files/garden.ron:7:19: error[SH2001]: Expected integer help[SH2001]: fix the file at the line and column given: the message says what was expected there ``` A field the type doesn't have, as `deny_unknown_fields` asked: ```text files/garden.ron:7:27: error[SH2001]: Unexpected field named `rain` in `Plan`, expected one of `plots`, `seeds`, or `grow_percent` instead ``` A number too big for its type, `plots: 300` in a `u8`: ```text files/garden.ron:5:15: error[SH2001]: Integer is out of bounds ``` A forgotten closing `)`: ```text files/garden.ron:10:1: error[SH2001]: Unexpected end of RON ``` The check `read_plan` makes by hand, which knows the file and not the line: ```text files/garden.ron: grow_percent is over 100 ``` And no file at all: ```text files/garden: error[SH2003]: file not found in asset bundle, nor garden.ron help[SH2003]: add the file at that path, or fix the name that asks for it ``` A letter the font atlas can't draw, `Łódź` in `en.ron`. The text is checked against the atlas once every file loads, and a language keeps its text by key, so the message names the key, not a line: ```text en: 'Ł' (U+0141) isn't in the font atlas (title.subtitle); remake the atlas with font-atlas from a font that has it ``` `cargo shuyi font-atlas files-src/ter-u16n.bdf crates/game/files/font` remakes it: it runs the game, and keeps every character its text uses that the font has. Japanese and Chinese characters are twice as wide as a letter: each takes **two cells** of the screen. They come from a second font, twice as wide (16 × 16 pixels beside 8 × 16), given with `--wide`: ```text cargo shuyi font-atlas --wide files-src/my-16x16.bdf files-src/ter-u16n.bdf crates/game/files/font ``` The atlas then keeps the wide font's glyphs of every character the text uses that the narrow font lacks, and of the characters `shuyi.ron`'s `font_atlas.wide_glyphs` names (a name entry's letters, say). Japanese and Chinese write many of the same characters with one code point but each in its own shape (直, 骨, 角). A game in both gives a wide font for each, as a named **set**: ```text cargo shuyi font-atlas --wide ja=files-src/ja-16.bdf --wide zh-Hans=files-src/zh-16.bdf files-src/ter-u16n.bdf crates/game/files/font ``` Each language pack names the set it draws from in its `lang.ron`, and a Chinese one says its quotes, ellipsis and dash (`“ ” ‘ ’ … —`) are wide, as Chinese writes them: ```ron (name: "简体中文", made_by: Machine, wide_font: Some("zh-Hans"), wide_punctuation: true) ``` The source language has no `lang.ron`: a game written in Chinese says the same in its code, where it declares the source: ```rust let source = SourceLang::new(zh.clone(), "简体中文") .with_wide_font("zh-Hans") .with_wide_punctuation(true); let lang = Lang::new(ui, BTreeMap::new()).with_source_lang(source); ``` A set keeps the characters of the languages that name it, and those `font_atlas.wide_set_glyphs` names for it (`{"ja": "..."}`). A wide character is drawn from the language's own set first, then from any other, so a pack's own name shows in its shapes whatever language is on; `cargo shuyi check` fails on a pack character its set lacks. A font in JIS X 0208, as many Japanese bitmap fonts are, needs `shuyi-cli`'s `jis` feature in the game's `tools`. Text measured for a layout counts a wide character as two: `shuyi::assets::lang::text_width` does, with the widths of the frame (`buf.widths()`), and `buf.text_cells(text)` gives the same as a column count. The checks that keep text to a length count so too, once they have the atlas's widths (`CellWidths::of(&atlas)`): a language pack's rules (`PackRules::widths`, or `LangPackRon::widths_of`), a tip's (`TipRules::widths`) and a dialogue's (`SceneRules::widths`). So load the font atlas before the text. The source's own text is measured in its language: `source.widths(&CellWidths::of(&atlas))`, or `lang.widths(code, &base)` for any language. ## Voices A language's voice clips sit in its voice folder, one per dialogue line, listed in its manifest (`voice::from_source`), and its cast says which voice says each speaker's lines (`voice::cast_from_source`). Whatever makes the clips, a speech program or an actor in a booth, is the game's own tool, and the engine tells that tool what to do: `voice::plan::plan` takes the dialogue, the scenes to voice, the cast and the manifest's clips, with three answers only the game knows: which clips a line needs (one, one per variant, or none for a reply nobody says aloud), the words a clip says, and whether a generated clip's voice changed (its sample was redone, say). It gives back every clip wanted, with the line before it, and which to make and why (`Why::Missing`, `Words`, `Direction`, `Recast`, or the tool's own reason), which a person must record again, which to delete, and which lines no voice is cast to say. A clip of a scene that isn't voiced is kept while its line is there. `voice::plan::recording_script(&plan, "guide")` lists one voice's lines in script order, for a recording session. It reads and writes no file, so the tool loads the files, plans, and does the work. ## Layers A bundle is **layers** read in order: a file comes from the first layer that has it. So a layer laid over the game's files replaces a file at its path, and adds the ones only it has. A player's **mod**, a change a player makes to a game, is such a layer; on desktop, `shuyi::app::read_folder` reads a folder on disk into one. On the web, where there are no folders, `shuyi::app::page::page_layer` reads the text files the game's page hands it (a page that edits the game's data beside it, and runs it again), and `page::report` tells the page what was wrong with them: load your data with the layer laid first, and lay it only if that works. ```rust #[test] fn a_mod_laid_over_the_games_files_replaces_its_plan() { let files = BTreeMap::from([( "garden.ron".to_owned(), b"(plots: 12, seeds: 20, grow_percent: 50)".to_vec(), )]); let layer = Layer::owned("mods/big-garden".to_owned(), files).unwrap(); // The first layer that has a file gives it. let bundle = Bundle::new(vec![layer, game().bundle().layers()[0]]); assert_eq!(read_plan(&bundle), Ok(Plan::new(12, 20, 50))); // Every other file still comes from the game's own layer. assert!(bundle.has("en.ron")); } ``` > **Rust:** `BTreeMap` is a list of values looked up by a key. `b"..."` is text as raw > bytes, which is what a file holds, and `.to_vec()` makes it a list you own. > `.to_owned()` does the same for a `&str`. A layer can also be private (files that may not be handed out, such as bought art: the tools say which files come from one), or hold only the tests' files, which a build that ships leaves out. ## A format of your own A file format the engine lacks is a **plugin**, something you add without changing the engine: a type that implements `Format` (its id, the extensions it claims, an optional sniff of its first bytes, a size limit, and `decode`), registered beside the others. It never panics on a broken file, says every problem rather than the first, and reads the same bytes into the same asset on every platform. The engine's [formats page](https://github.com/Zafnok/shuyi-engine/blob/main/docs/formats.md) walks through one, with its tests, and lists the built-in formats. ## Settings beside a file Some choices differ from one file to the next: how a picture is made one-bit, which palette it uses. They go in a small RON file beside it, `images/hero.png.meta`, or for a whole folder in its `_folder.meta`. Each holds sections named after the step that reads them, such as `(onebit: (dither: bayer))`, and the nearest file with a section wins. A format reads its section with `DecodeCx::import_settings`, and a typo in any settings file is an error with its line. The formats page says [how settings are found](https://github.com/Zafnok/shuyi-engine/blob/main/docs/formats.md#import-settings). ## Reloading while the game runs Waiting for a rebuild to see a new color gets old. `cargo shuyi run --watch` starts the game in its window, in a debug build, with the folder of its files on disk laid over the embedded ones (`assets:` in `shuyi.ron` names it). Save a file there, and within a second the bundle has its new bytes. The pictures the window draws, the font's and the images', are redrawn from them with no code of yours; a sprite keeps its source rectangle, so one past the edge of a smaller picture shows as missing, in magenta. A value you want to follow it is a `Hot`: any kind, read through the registry, which remembers every file its decode asked for (its picture, its settings) and reads like the value. A small module of the game's keeps it, and refreshes it when files change: ```rust /// The garden's plan, reloading as its file changes. A screen reads it from /// the context, as it reads any module's state. pub struct HotPlan { /// The plan: reads like a `Plan`. pub plan: Hot, } impl HotPlan { /// The plan of `bundle`, or every problem with it. pub fn new(bundle: &Bundle) -> Result> { let plan = Hot::load(&Arc::new(formats()), bundle, "garden")?; Ok(Self { plan }) } } impl Module for HotPlan { fn name(&self) -> &'static str { "hot_plan" } fn install(&mut self, install: &mut Install<'_, G>) { install.hook(Hook::Reloaded); } fn reloaded(&mut self, ctx: &mut Ctx, changed: &[String]) { // A broken file keeps the old plan, and shows its error. let reload = self.plan.refresh(changed); ctx.note_reload(self.plan.id(), &reload); } } ``` > **Rust:** `impl Module for HotPlan` says `HotPlan` is a module of any > game: `G` is a blank, and `G: GameData` says it must be a type with that trait. `Arc` > lets several owners share one value. `'_` lets the compiler work out how long a borrow > lasts. A broken file never stops the game: the old value stays, and `note_reload` shows its error, with its file and line, at the top of the frame until the next good save. A reload is no input and is in no record, and a build that ships never looks at the disk. It isn't on the web, which rebuilds the page instead, or on a console. ## The pack A game may ship its files **baked** into one pack, a `.shuyipak` file indexed for fast loading. `cargo shuyi bake` makes it, and `cargo shuyi check --baked` proves every asset read from the pack equal to the one read from its source, so the text files stay what the game is made of. The formats page says [how a pack is built](https://github.com/Zafnok/shuyi-engine/blob/main/docs/formats.md#the-pack). ## A target's own files A **target** is a kind of machine a game runs on. A folder `assets@playdate/` beside `assets/` holds the files the Playdate uses in place of the game's: a picture drawn by hand in black and white, a settings file. `Bundle::for_target` lays it over `assets/` when the game bakes or runs for that target (`ctx.platform.target`), and leaves every other target's out. On a black-and-white screen a game bakes every picture once as a one-bit picture (`Bake::under_as` with `OneBitFile`), so the device never **dithers**, the job of turning colors into patterns of black and white dots. Each PNG is dithered by its `onebit` settings: `atkinson` for pictures, `bayer` for flat art and anything that moves, `threshold(N)` for line art, or `none` for art already black and white, unless the target's folder has its own. `cargo shuyi bake --target playdate --sheet sheet.png` names the pictures it dithered and draws each beside its source, so you see which ones are worth drawing by hand. The formats page says [what each dither does](https://github.com/Zafnok/shuyi-engine/blob/main/docs/formats.md#the-onebit-section).