Repository navigation
missing termios.h functionality #161
Description
while trying to port a simple command line application to WASI and the console crate in particular, i stumbled over some unpleasant limitations concerning unbuffered keyboard input and suppressed echo output for passwd entries as required by many CLI applications.
i perfectly understand the motivations, why this features were not needed/available in CloudABI, and i respect the reservations to pick up a POSIX interface, which is full of unnecessary historic cruft and furthermore isn't supported on the windows platform, nevertheless there is not much alternative available except inventing the wheel once again. but without any intermediating system interface, we are simply not able to work around this grave limitation.
in practice we would most likely only need a small part of the features provided by termios.h resp. tcgetattr/tcsetattr/tcflush, but it's perhaps better for the practical portability of existing *nix code to just reuse this old fashioned POSIX interface the way it is. this should work on nearly any platform except windows, but even on this platform we could implement the needed features via SetConsoleMode etc. without much security worries.
the most essential feature missing are IMHO:
- unbuffered single key input
- suppressed echo for password entries
most other terminal control is anyway available via virtual terminal escape sequences (...although a more suitable forwarding of environment variables by the runtime implementations -- especially: TERM, TZ and the locale related ones -- would be also very helpful for this purpose resp. better CLI application support).
there are already similar closed feature requests available (eg. #42) and some brave developers are already looking for rather unorthodox workarounds (see: wasmerio/wasmer-sdk#141), nevertheless i really would like to get an idea, what we should expected as the desired solution for this issue in the long run.
Activity
sunfishcode commentedon Nov 27, 2019
Thanks for reporting this! The current practice of assuming programs can use ANSI-style escape codes is actually problematic from an ecosystem perspective because the functionality differs between platforms. It'd be great to work together to find solutions here.
These seem like useful features
For TZ and LC_*/LANG/LANGUAGE
dumblob commentedon Nov 27, 2019
Feel free to look how such highly cross-platform functionality is practically implemented in neovim forks which uses LeoNerd's libtermkey and libvterm for that.
mash-graz commentedon Nov 28, 2019
thanks @sunfishcode for this warm words of welcome!
when i stumbled over this issue, i thought at first, i could prepare a sketch of a solution for this problem, but after spending a few days of studding the requirements, i came to the conclusion, that it is in fact far beyond my actual capabilities. nevertheless it's perhaps an interesting case, to debate the actual limitations of WASI and look for desirable solutions to workaround the given situation in an exemplary manner.
in one or the other way it seems to be necessary to extend the actual state of WASI to realize a solution for this issue. and that's indeed a serious endeavor, which shouldn't be handled in an unreflected ad hoc manner. unfortunately it's also affecting many different components -- i.e.: the WASI core interfaces/specification, WASI libc suport and the actual used runtimes --, so it's rather hard to even play around with experimental drafts of improvement code.
it may be also seen as an example, where Optional Imports resp. modularization could make a lot of sense.
e.g. for browser based runtime utilization or minimalist embedded IoT scenarios it's simply not necessary or useful to provide this kind of features and make everything bloated. but then again it's nearly indispensable for satisfying CLI application support.
platform specific handling
solving this particular requirement is a very tricky challenge, because it's by no means trivial to find a solution, which satisfies traditional POSIX
termios.hutilizing system just the same as the windows platform by one single mechanism. the fundamental differences on both sides are very hard to overcome. in nearly all existing cross platform code this hindrance is solved by conditional compilation of different code segments. but that's a solution, which isn't compatible with WASIs general concept. we therefore have to chose between two other options:or
for good reasons some of us may prefer a.) and there is already a multitude of intermediate terminal handling libraries and language runtimes available, which realize this approach. unfortunately many of them share the the same unpleasant characteristics: they all use different functions and control concepts in their wrappers, usually support only a small part of the actual system capabilities and tend to be very constricting and error prone in practice.
although i would always use one of the available mature intermediate terminal handling libraries for complexity reduction in case of practical user level development, because the native system interfaces are often hard to utilize in an economic manner, i still prefer variant b.), because it still preserves full access to all system specific capabilities in principle and makes it much easier to port existing code resp. replace the intermediate layer by some other library/crate whenever you like.
but it's indeed a puzzling question, which of this two approaches makes more sense? the pros and cons of both alternatives have be respected. but at the end it shouldn't make a big difference, which variant we choose, as long as the solution is realized in a modular fashion, which can be further improved over time.
general reservations concerning terminal feature utilization
although i agree with with all your efforts to reduce the WASI core interface to the absolute necessary minimum, keep it secure and platform agnostic, i wouldn't see virtual terminal utilization as an hostile enemy and avoidable source of incompatibility. quite the opposite! in fact it provides a set of features, that would otherwise need system specific control interfaces and vendor specific obscurities. e.g. utilizing CSI-controll we don't need additional system specific functions to query the terminal size and cursor position, because this kind of information can be received in a rather system independent manner by just utilizing common in-band terminal communication whenever VT handling is enabled (e.g. by ENABLE_VIRTUAL_TERMINAL_PROCESSING). for a long time this worked only reliable on *nix machines, but in the meanwhile even windows boxes obtained remarkable improvements regarding this capabilities (see e.g.: https://devblogs.microsoft.com/commandline/).
it's also slightly misleading, if we associate more advanced terminal handling only with unnecessary colored output, emoji support and other fancy cosmetic crap. just think about the usefulness of readline command completion resp. history access. in this case you definitely don't want want to be forced to always press an additionally ↩️ key after every every ⬆️ keypress, just to recall the last command e.g. in a minimalist pure text mode debugger. and that's more the kind of usage, which i'm actually looking for. it's also a very simple example to illustrate, what we can't realize right now with WASI. if toggling un/buffered terminal mode and echo suppression would be possible via common escape sequences as well, i would immediately utilize this workaround and avoid to bother anybody with my desires.
conclusion
i'm sure, there are much more important tasks waiting to be solved by the WASI core developers, but perhaps this particular feature request could also serve as a demonstration, how this kind of improvements can be solved by a modular approach. such an comprehensible example would be very instructive for many similar cases. but that's perhaps just an utterly utopian fantasy.
i hope, you can excuse my lengthy elaborations and grasp at least some inspirations out of all this confuse rubbish.
dumblob commentedon Nov 28, 2019
@mash-graz this is a great writeup, thanks!
I would emphasize the following from your post as it leads me to the question "What everything could be moved from WASI to the potential
LanguageTerminal Server Protocol?":mash-graz commentedon Nov 28, 2019
@dumblob
thanks for your feedback!
and yes, this alternative would be also possible!
in fact it's a very old problem, how terminal communication can be serialized in a manner, which doesn't need any out of band control channels or direct API access.
telnet (RFC854) is such an example. it's in fact a transport layer agnostic protocol and could be used in principle also over very simple stdio communication. it's more oriented towards line buffered communication, because of it's performance benefits, but nevertheless it also provides commands to utilizes single character operation. it also comes with clear specified options for echo handling (RFC857), terminal-type identification (RFC1091), environment variable forwarding (RFC1572) etc.
so we could indeed use this protocol to tunnel al the needed features over the already existing WASI stdio infrastructure in a platform independent manner and just translate the control commands within the runtime on the host side to system specific API calls. but for our applications we would still have to provide some en/decoding facilities resp. interface again, to make this capabilities accessible. therefore i doubt, that all this de/serialization efforts will bring about any advantage in our case.
communication with the outside world isn't the crux of our problem and we should better avoid the performance penalties of unnecessary serialization overheap. we only have to look for a minimalist and efficient interface, which still satisfies the specific needs of the divergent native POSIX and windows APIs for the desired purposes.
dumblob commentedon Nov 29, 2019
My point is, that we could leverage such "inner protocols" for problems which won't have any security implications and thus don't need to be part of the capabilities API. Therefore I'm afraid we can not "just reuse" an existing protocol as they're not designed with security in mind. But for some functionality it might make sense.
Security it pretty complicated thing - imagine that e.g. turning on colors could make an existing system vulnerable to phishing etc. by using similar looking colors to the default black&white etc. - but that's exactly the discussion we need to have first and decide up to which degree the capabilities should touch on such "inner protocols".
mash-graz commentedon Nov 29, 2019
well -- security is always a compromise between useful limitation and preserving the actual capabilities resp. freedom of computing.
in this particular case i'm definitely convinced, that it doesn't make much sense to combat the dangers by any overly restrictive WASM design. in this respect it will hardly become better than the POSIX design, as long as we want to preserve a satisfaying base for more demanding command line applications -- e.g. editors.
the POSIX
termio.hinterface was already an specification attempt, to make the necessary calls more save and platform agnostic resp. portable than the underlyingioctl_ttycalls.as a long time linux user i'm well aware of all this [less] funny tricks to abuse terminal control sequences etc. nevertheless i respect and share the opinion of most security experts, that this risk can only be effectively solved by improving the utilized virtual terminal applications. that's the sole relevant place, where this kind of abuse can be prevented in a really consequent and reliable manner. but trying to handle it here in WASI is IMHO a hopeless ineffective endeavor.
and i really have to comment, that i want to use WASI for serious work, and not a just to play around with an utterly crippled toy, which isn't anymore usable e.g. for more powerful
ncursesorreadlinebased CLI software.sunfishcode commentedon Dec 3, 2019
@mash-graz
Yeah, there are several related topics. I've followed up on ANSI escape sequences and terminal input in the separate issues I've now opened for them so we can discuss those separately. Also, I apologize for accidentally posting an incomplete comment above. To finish my thought, I agree that locale and timezone info are useful, so I've now filed #167 to track that.
I agree that enabling unbuffered input and echo suppression are useful and pretty simple and we can probably do them without complication.
@dumblob
Computing is full of "everything is a"s, isn't it ;-). It's great to bring ideas into the discussion; we also need help digesting them into actionable form.
mash-graz commentedon Dec 3, 2019
this would be a really useful improvement!
it would indeed solve the requirement, which i originally couldn't work around.
and thanks for all your efforts to redirect the related topics in more useful individual tickets.
bryzaguy commentedon Feb 19, 2021
This would be incredible! Especially if this led to an alternative to readline outside of WASI. I'm building a REPL and am shocked there aren't better, more widely supported solutions. I would leap at the chance to have people coding in my language in the browser.
bryzaguy commentedon Feb 19, 2021
For context, a buffered solution doesn't allow you to do magical things like autocomplete as you type which feel really good.
LinusU commentedon Oct 11, 2022
I ran into this whilst trying to compile my small CLI application to WASM/WASI. Does anyone know if there has been anymore progress on this outside of this issue?
sunfishcode commentedon Oct 12, 2022
A while ago I started putting together a design related to this, which is documented here:
https://basic-text.sunfishcode.online/Terminal.html
It's not yet in practical shape, but it might be interesting as a starting point for discussion. Some thoughts;
Puellaquae commentedon Nov 26, 2022
It looks good. I'd to know whether it will become a proposal of wasi, and whether it's planned to provide some function to get info from terminal, such as terminal's size.
TristanCacqueray commentedon Jun 10, 2023
It's a bit unfortunate WASI doesn't support unbuffered input that are necessary for interactive terminal games. Is this going to be ever be implemented in WASI, or should we look into WASIX which seems to handle TTY ?
sunfishcode commentedon Jul 20, 2023
Building on the ideas in #162 and https://basic-text.sunfishcode.online/Terminal.html, and upcoming versions of wit-bindgen with handle support, I now have a draft PR here filling in the interfaces for
terminal-inputandterminal-outputwith an API that should support the most important terminal features, and it even gives some descriptions of escape codes which are expected to be supported.A difference with the earlier approaches is that this doesn't require all implementations to parse and translate the bytestream. The escape codes described should have pretty consistent behavior across popular terminals. This has the downside of having nothing to prevent applications from having non-portable behavior, but this is for Preview 2, and we can re-evaluate that as WASI matures.
That said, implementations may still wish to parse the bytestream if they wish more control about which escape codes can be used or how they're interpreted, such as to control which information can be reported from the terminal back into the input of the application, to control how the output of an application can influence the output of other applications, or other things.
Feedback welcome!
Merge pull request WebAssembly#161 from idpbond/patch-1
ricochet commentedon Sep 3, 2026
We discussed this in triage. We believe this is very interesting and needs someone to build on the initial sketches made by @sunfishcode against the current wasi-cli interface. We welcome anyone who is interested to step in and drive this forward.
lann commentedon Sep 3, 2026
See: WebAssembly/wasi-cli#62