The Modern Embedded Developer Toolchain, from SDK to Hardware-in-the-Loop

Laptop showing source code beside a notebook on a developer desk Photo by Kevin Ku / Pexels

The GitHub check is green. The firmware does not boot.

I have watched this exact failure on a bench more times than I want to admit. The host unit tests mocked the UART. The linker map looks fine. The pin that actually matters is remapped in a Devicetree overlay that only exists on the real board. OTA was never exercised. Someone still has to walk to the lab, plug in a probe, and find out what CI never saw.

That is not an IDE problem. It is a pipeline problem. The modern embedded toolchain is the path from SDK install to a hardware-in-the-loop merge gate. If any layer is a snowflake on one engineer's laptop, the board will teach you about it after the pull request lands.

Key Takeaways

Among developers running an RTOS, 51% cite hardware and platform support and 42% cite developer tooling as RTOS selection factors (Eclipse IoT & Embedded Survey, 2025).

On constrained devices, C remains 55%, C++ 36%, with Zephyr at 21% and 75% of respondents using open source (Eclipse survey, 2024).

The HIL testing market grew from $1.12 billion in 2025 to $1.23 billion in 2026 (The Business Research Company, 2026).

Renode now reports 926 of 1,513 Zephyr platforms passing Hello World, up from about 100 when the dashboard launched (Antmicro, Aug 2026).

PlatformIO's VS Code extension sits at 7.2 million installs. ESP-IDF 6.0 (March 2026) added CMake presets, EIM, and a built-in MCP server.

The Toolchain Is Six Layers, Not One IDE

Vendor "getting started" pages still sell a single installer. In 2026 the work actually spans six layers that have to agree with each other:

Editor and SDK: VS Code plus a vendor pack that can install a pinned toolchain
Build: CMake, Ninja, west or idf.py, with presets instead of tribal flags
Quality: host unit tests, static analysis, and a second compile for the real target
Debug: a probe that speaks DAP, plus RTT or a serial harness you can script
Emulation: QEMU for boot and scheduler smoke, Renode when peripherals matter
HIL: flash a real board from CI and fail the PR if the device misbehaves

Skip a layer and you get a local hero workflow: it works on the desk that owns the J-Link, and nowhere else.

Eclipse's 2025 survey is blunt about why this now shows up in tool selection, not just taste. Top development challenges were data collection and decision-making at the edge (35%), maintenance and lifecycle updates (24%), and deployment complexity (19%). Cybersecurity was 17%. Time-to-market was 16%. When those teams do pick an RTOS, developer tooling (42%) sits right behind hardware and platform support (51%). The SDK that cannot be installed the same way in CI as on a laptop is already a schedule risk.

Layer 1: The SDK Lives in the Editor Now

VS Code is the hub. The interesting split is which vendor pack owns the toolchain.

Espressif. ESP-IDF 6.0 shipped in March 2026 with the ESP-IDF Installation Manager (EIM): one installer for GUI, CLI, and CI, with a GitHub Action and no more export.sh ritual. The May 2026 VS Code extension v2.1.0 made EIM the default even in WSL, Dev Containers, and Codespaces. v1.11 added a native Unity Test Runner and Copilot commands that can rebuild from chat. v6.0 also ships a built-in MCP server (eim run "idf.py mcp-server") so an agent can build, flash, and inspect connected devices without inheriting a broken shell environment.

Nordic. nRF Connect for VS Code is the recommended path into the nRF Connect SDK. West still owns the multi-repo workspace. The first LTS line (v3.4.0) is the one you pin if you need years of security patches instead of the newest experimental SoC.

ST. STM32CubeIDE for Visual Studio Code landed in May 2025 and grew a March 2026 wave of CMake import tools: convert an Eclipse project, pin GCC or LLVM per project, and stop treating .cproject as the source of truth.

PlatformIO. Still the multi-vendor escape hatch: Apache 2.0, 1,000-plus boards, 40-plus platforms, and 7.2 million VS Code installs as of this writing. Use it when you hop architectures. Pin the platform version in platformio.ini. Letting the wizard float to "latest zip" is how two laptops compile different firmware from the same commit.

If your team cannot reproduce a toolchain with one documented command, you do not have a toolchain. You have folklore.

Layer 2: Builds Have to Be Boring

West and CMake are the lingua franca for Zephyr-shaped trees. ESP-IDF still wraps CMake with idf.py. Ninja should be the generator everywhere you can choose it.

The 2026 habit that actually saves weekends is named presets, not longer wiki pages. ESP-IDF 6.0's CMake presets turn this:

idf.py -B build_prod -D SDKCONFIG_DEFAULTS="sdkconfig.defaults.production" build

into this:

idf.py --preset production build

Do the same with west:

west init -m https://github.com/zephyrproject-rtos/zephyr --mr v4.1.0
west update
west build -b nrf52840dk/nrf52840 app --pristine

Pin the west manifest revision, the SDK tag, and the host packages. Commit CMakePresets.json or .vscode/zephyr-ide.json. Do not commit build/ or a machine-local sdkconfig that drifted from sdkconfig.defaults.

Qualcomm's PMIC software team made this concrete at Open Source Summit North America 2026. They had two firmware stacks: proprietary RTOS on production silicon, a different RTOS on a pre-silicon FPGA, separate repos, and a porting tax on every bug fix. Moving production to Zephyr let one application ride Devicetree and Kconfig across both platforms, with Twister plus Pytest as the HIL gate (Zephyr Project write-up, July 2026). The build system was the unification, not a dashboard.

Layer 3: Quality Before You Touch Silicon

Host tests are not optional theater. They are how you get coverage into the dark corners a UART harness will never reach.

The two-compiler pattern still wins:

Host GCC or Clang compiles Unity (or GoogleTest) suites with CMock-generated doubles
arm-none-eabi-gcc or the vendor clang compiles the real ELF so the target still links

Ceedling remains the practical orchestrator for C. ESP-IDF's VS Code Unity runner is the same idea inside one vendor tree. Put cppcheck on every PR. Run clang-tidy or the Clang Static Analyzer on a nightly. Reserve a qualified MISRA tool for the release branch if you actually have to show ISO 26262 or IEC 61508 evidence. Cppcheck is not that evidence.

Language choice is now a security decision, not a style fight. VDC Strategy's survey of 500-plus embedded decision-makers found 85% saying programming language choice directly affects safety and security (AdaCore / VDC, 2026). C is still the default on constrained devices. That does not excuse skipping sanitizers on host tests or treating "we reviewed it" as a quality system.

A merge should fail if host tests fail, if the target does not link, or if ERROR-level static analysis is new. "We will test on the bench tomorrow" is how OTA bricks a fleet.

Engineer assembling electronics at a workbench with a soldering iron Photo by ThisIsEngineering / Pexels

Layer 4: Debug Like the Editor Is the Debugger

printf over UART is a sensor, not a debugger. You still need halt, watchpoints, and a stack unwind when the fault is in a startup path that never reaches the console.

probe-rs is the 2026 default I reach for first on ARM, RISC-V, and Xtensa: DAP in VS Code, CMSIS-Pack flash algorithms, RTT, and a CLI (probe-rs run) that does not require a 200-line OpenOCD config. It has about 2,860 GitHub stars and a crates.io crate seeing tens of thousands of downloads a month. OpenOCD and vendor GDB servers remain the fallback when a pack or a weird TAP is missing.

Wire the probe through Debug Adapter Protocol. Cortex-Debug, nRF Connect, ESP-IDF's Eclipse CDT adapter, and probe-rs all speak it. The engineer who can only debug inside one vendor Eclipse install is a bus factor of one.

Script the boring path too. A CI job that dumps RTT or serial until a banner appears is worth more than a beautiful launch.json that nobody runs.

Layer 5: Emulation Is a Merge Gate, Not a Demo

QEMU is the cheap smoke test: does this image boot, print, and exit? Renode is what you use when the failure is a peripheral, a multi-node bus, or a scripted sensor.

Antmicro's Renode Zephyr Dashboard (August 12, 2026) is the existence proof that this scales. They started with about 100 passing platforms. As of that post, 926 of 1,513 Zephyr platforms pass Hello World. The U-Boot dashboard is 829 of 1,373 (60%). dts2repl generates Renode platform files from Devicetree, which is the same reason Zephyr travels across boards in software.

Put west twister on QEMU in cloud CI. Add Renode via renode-test-action for the boards you actually ship. Do not pretend a passing QEMU run means the ADC driver works.

Layer 6: Hardware-in-the-Loop Means the PR Touches a Board

A HIL test is a system test on real silicon, with real peripherals, driven by a host that can fail the merge. Unit tests raise coverage. HIL raises confidence. You need both. 100% coverage on mocks can still ship a device that never joins Wi-Fi.

Golioth documented the budget version years ago and it still holds: one engineer, one week, a Raspberry Pi as a GitHub Actions self-hosted runner, boards on USB, firmware built in the cloud, flash and serial verify on the Pi (HIL part 1, part 2). Zephyr Twister on a hardware map is the same pattern without writing your own runner:

west twister --device-testing --hardware-map map.yml -T tests/

Qualcomm's OSS NA 2026 setup went further: Twister compiled for a Raspberry Pi Pico, flashed it, then Pytest toggled a USB-C switch on a PMIC eval kit and asserted VBUS through plug cycles. The job was a merge gate. Logs lived as GitHub artifacts so you did not need a badge to walk into the lab.

The commercial HIL market is not a hobby. The Business Research Company puts it at $1.12 billion in 2025 and $1.23 billion in 2026 (10.6% CAGR), with Emerson's March 2025 Embedded Software HIL Test Suite as the software-defined end of that world. Better Devices' 2026 field guide is the practical ladder: under $1,000 to start, $5,000–$20,000 for a mid-range industrial bench, $50,000-plus when you actually need NI, dSPACE, or OPAL-RT determinism.

If your "HIL" is a board on someone's desk that is unplugged every Friday, it is not CI. Next week's post in this series is about turning that bench into shared lab infrastructure. This post is the contract the firmware side has to keep: build in the cloud, flash on a labeled runner, parse a real serial log, block the merge.

Close-up of a computer motherboard with chips, traces, and capacitors Photo by Luis Gomes / Pexels

A Weekend Starter Stack

Do not boil the ocean. Wire one product path.

Friday night: Install the vendor VS Code pack (ESP-IDF + EIM, nRF Connect, or STM32 CMake). Pin the SDK. Confirm west build or idf.py --preset dev build works from a clean clone.

Saturday: Add Unity tests for the state machine you keep breaking. Run them on the host in GitHub Actions. Add cppcheck. Fail the job on ERROR.

Sunday morning: Twister on QEMU for hello_world. Sunday afternoon: one self-hosted runner, one DK, one hardware map, one test that asserts a boot banner and a GPIO or a cloud ping. Build in the cloud. Flash on the runner. Keep the probe plugged in.

Measure three things: time from push to a board result, number of "works on my desk" incidents per month, and whether OTA is in the HIL path at all. If OTA is still a click-through ritual, you are not done.

What Still Breaks

Honesty matters here.

Flaky HIL is worse than no HIL. A yellow board, a busy serial port, or a probe that enumerates as COM17 on Tuesdays will train the team to ignore the gate. Lock USB paths by by-id, power-cycle from a script, and quarantine a failing fixture instead of retrying until green.
Emulators lie about analog, RF, and timing. Renode will not catch a cracked crystal or a layout that couples into ADC. That is why layer 6 exists.
Vendor IDEs still own some packs. TrustZone dual-core STM32 imports, some proprietary compilers, and certified compiler qualification kits are not VS Code problems yet. Isolate those behind a pinned container rather than making every laptop match.
Toolchain drift. Two EIM versions, two west manifests, and a floating PlatformIO platform will recreate the "green CI, dead board" bug in software. Pin everything, including the runner image.

The goal is not to replace the engineer who can read a scope. The goal is to stop spending that person on "did we even flash the binary we committed?"

If you are already connecting this stack to agents, last week's post on MCP as the integration layer is the contract on top. The toolchain still has to be deterministic underneath. Agents cannot debug a toolchain you cannot reproduce.

Next in this series: hardware CI, and how a lab bench becomes shared infrastructure.