“Ghost in the DCL Shell: OpenVMS, Touted as Ultra Reliable, Had a Local Root Hole for 30 Years”
John Leyden, The Register, February 6, 2018
VMS uses four modes: user mode; supervisor mode, where the DCL [Digital Command Language] shell runs; executive mode for privileged services; and kernel mode, which has power over the system.
VMS runs its shell in supervisor mode. A program can pass malformed command line data to DCL to process, which overflows a buffer and clobbers a return pointer in memory. There are some portions of memory with fixed addresses that all programs which run in a process share, and for some reason can hold executable code. Thus, it's possible to stash some malicious code in those shared areas, pass a booby-trapped command line to the shell to parse, and have the shell jump to the evil attacker-controlled code while still in supervisor mode. …
Furthermore, … the boundary between supervisor and executive mode is not as watertight as folks are led to believe. Thus, it is possible to leverage the escalation from user mode to supervisor mode to jump into the executive and drill deeper into the system.