You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Aug 17, 2022. It is now read-only.
Repository navigation
This repository was archived by the owner on Aug 17, 2022. It is now read-only.
STRING | Converts the next two arguments from a pair of i32s to a utf8 string. It treats the first as an address in linear memory of the string bytes, and the second as a length.
Any chance that there'll be support for JS-style strings (UTF-16LE) as well? I know this doesn't really fit into the C/C++ world, but languages approaching things the other way around will most likely benefit when not having to convert back and forth on every host binding call.
Since this hasn't received any comments yet, allow me to bump this: I am still curious if UTF-16 strings can be supported. In AssemblyScript's case all strings are UTF-16LE already, so only having the option to re-encode (potentially twice if the bound API wants UTF-16) does seem like it should be taken into account.
This specific operator has not been discussed. In part that is because we
are still at a more 'skeletal' level in the effort.
However, although definitely possible, the bar for coercion operators is
going to look something like this:
a. Is the operator consistent with the requirements of the majority of
hosts (i.e., mostly browsers at this point but with a definite leaning
towards non-browser implementations)?
b. Is the operator consistent with the majority use case for representing
(in this case) string values?
If most hosts would require copying unicode16 into urf8 anyway, you may
have trouble with (a).
But, essentially, it's a bit early to consider committing to any operators
at the moment.
On Sun, Jun 30, 2019 at 1:56 AM Daniel Wirtz ***@***.***> wrote:
Since this hasn't received any comments yet, allow me to bump this: I am
still curious if UTF-16 strings can be supported. In AssemblyScript's case
all strings are UTF-16LE already, so only having the option to re-encode
(potentially twice if the bound API wants UTF-16) does seem like it should
be taken into account.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#13>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAQAXUEQUF23TJ7APNWB5ODP5BYNVANCNFSM4EJGLDYA>
.
If most hosts would require copying unicode16 into urf8 anyway, you may
have trouble with (a).
To me it looks like having such an operator can lead to significantly less work where UTF-16 is already present on both sides of the equation, while implementing the APIs in any case where either side is UTF-8 can easily be implemented by reencoding conditionally. Hence, the module would chose the operator that fits its internal string layout ideally, and the host would do whatever is necessary to make it fit into theirs. This leaves us with those cases:
Both UTF-8: Essentially memcpy
Both UTF-16: Essentially memcpy
One UTF-8, the other UTF-16: Reencoding once
while avoiding the very unfortunate case of
Module UTF-16, host UTF-16: Reencode twice because UTF-8 is all the bindings understand
But, essentially, it's a bit early to consider committing to any operators
at the moment.
I see, yet I thought it might make sense to raise this early so once committing to operators, this case is well thought through :)
Reacted by Volker Berlin, Oskar Larsson Högfeldt, Volodymyr Shymanskyy, Amin Ya and Jeff Hykin
Module UTF-16, host UTF-16: Reencode twice because UTF-8 is all the bindings understand
Yeah that would be unfortunate.
Any chance that there'll be support for JS-style strings (UTF-16LE)
The real question is, can we skip JS entirely? At which point, what does the host API use internally? For example a similarly bad outcome would be
Source UTF-8 -> JS UTF-16 -> Web API UTF-8
So I think the sanest way to handle that is with a declarative API on the bindings layer. Which is what you said earlier:
Hence, the module would chose the operator that fits its internal string layout ideally, and the host would do whatever is necessary to make it fit into theirs.
So the higher-level point is that we should be able to adequately describe the most common/reasonable ways to encode strings, so that we can minimize the number of encodings in the best case.
But, essentially, it's a bit early to consider committing to any operators
at the moment.
Agree + disagree. On the one, it's all in the sketch stage at the moment, where we're feeling out the rough edges. So from a managing-expectations point of view, this makes sense to say.
On the other hand, it's kind of incongruous to say "everything's up in the air, so don't raise any design issues." I don't think that was the intent, but that's kind of how it sounded. A more real translation of how I heard it was "don't worry about this now, we'll figure it out later." To which I would say, as a general principle, that yes we'll figure it out later, but we should raise it now to figure out if we should worry about it now. Especially because multiple people can think about different bits of the spec asynchronously.
I find it incredibly likely that we will default to 1 binding expression per ⛄-type per wasm representation (e.g. 1 for linear memory and 1 for gc), which is to say the MVP of ⛄-bindings will have one binding expression per type, because gc will probably not be shipped yet. On that basis, we will probably start with only UTF-8-encoding (I imagine we will drop the utf8-cstr binding too, for similar reasons).
My general mental model here is that we can always add bindings in the future as we find a need for them. And it may be the case that in practice, the re-encoding from UTF-16 isn't enough of a bottleneck to be worth it. Unless it is, at which point we can add that binding, and it will be more obviously useful because we'll have much more real-world data.
Also for AssemblyScript specifically, would it be reasonable to change the internal string representation from UTF-16 to UTF-8 in the presence of ⛄-bindings? It is, after all, "Definitely not a TypeScript to WebAssembly compiler" 😄
And it may be the case that in practice, the re-encoding from UTF-16 isn't enough of a bottleneck to be worth it. Unless it is, at which point we can add that binding, and it will be more obviously useful because we'll have much more real-world data.
At the end of the day we are building just tools here and one can't know the use case of everyone. Like, any use case extensively calling bound functions with string arguments would hit this and my expectation would be that this'll happen anyway (in certain use cases). Like, if we'd wait, this'll surface sooner or later, so it can as well be addressed from the start, instead of having to tell everyone running into this that their use case is currently not well-supported even through we did see it coming. Especially since specification and implementation of new operators can take a long time again.
Also for AssemblyScript specifically, would it be reasonable to change the internal string representation from UTF-16 to UTF-8 in the presence of ⛄️-bindings? It is, after all, "Definitely not a TypeScript to WebAssembly compiler" 😄
I'm sorry, the "⛄️-bindings" term is new to me. Would you point me into the right direction where I can learn about it? :)
Regarding UTF-8: In fact we have been thinking about this but it doesn't seem feasible, because we are re-implemeting String after the JS-API (with other stdlib components relying on it) and going with something else than UCS-2 representation seems suboptimal since the API is so deeply rooted into the language that mimicking UCS-2 would cost too much perf-wise. After all we are trying to stay as close to TS as reasonable to make picking up AssemblyScript a smooth experience. Also would like to note that this isn't exclusively an AssemblyScript thing, as other languages are using UTF-16LE as well, like everything in the .NET/Mono space.
Like, if we'd wait, this'll surface sooner or later, so it can as well be addressed from the start, instead of having to tell everyone running into this that their use case is currently not well-supported even through we did see it coming. Especially since specification and implementation of new operators can take a long time again.
It's ultimately a tradeoff. My thoughts here are that it will be strictly easier to spec and implement a bindings proposal that defines 8 operators, as opposed to one that defines 40. So we could just add UTF-16, but we could also just add C-strings and we could just add Scheme cons-list strings and we could just add Haskell lazy cons thunks, and so on. So for MVP I think we need to be really strict as to what exactly is "minimal", and in this context Minimal means "we can reason about strings at all".
We also need to balance the "viable" portion. Originally I was thinking we should avoid reasoning about strings and allocators at all, due to the complexity they add. Further discussion on this (see: #25) made me realize that not having an answer for allocators would compromise the viability of the proposal entirely. On that basis, not having UTF-16 support from day 1 is unlikely to leave the bindings proposal dead in the water.
By means of analogy, I would rather we ship anyref without waiting for the full gc proposal, because anyref on its own is a very enabling feature. It is in many ways suboptimal, but it it is more useful than what we had before. On that basis, I want to be very cautious about adding scope to the bindings MVP, especially when that scope is separable to a v2 that describes an expanded set of binding expressions.
I'm sorry, the "snowman-bindings" term is new to me. Would you point me into the right direction where I can learn about it? :)
tl;dr does this wasm binding layer we're describing need to reason about WebIDL at its core, or is WebIDL another target with a produce/consume pair? If the latter, and we suspect that is the case, then we're free to design an IDL that better matches what we're trying to do, rather than try to retrofit that on top of WebIDL.
Also would like to note that this isn't exclusively an AssemblyScript thing
Didn't mean to sound like I was saying it was :x, sorry. I was thinking that if AssemblyScript was using UTF-16 for easier FFI with JS, then in the presence of something-bindings it would be possible to decouple that ABI. And also that AssemblyScript would probably have an easier time of making that ABI switch than a more-ossified target like .NET, on account of it's a younger platform.
My thoughts here are that it will be strictly easier to spec and implement a bindings proposal that defines 8 operators, as opposed to one that defines 40
Makes sense, yeah. Though, to me it seems not overly complex to have a (potentially extensible) immediate operand on str (/ alloc-str) that indicates a well-known encoding. I'd consider UTF-8, UTF-16LE and maybe ASCII here (not sure), with length always provided by the caller (even if null-terminated), but I'm certainly not an expert in this regard.
By means of analogy, I would rather we ship anyref without waiting for the full gc proposal, because anyref on its own is a very enabling feature. It is in many ways suboptimal, but it it is more useful than what we had before.
I totally agree with the anyref mention, but don't entirely agree on the comparison with encodings. anyref is a useful feature on its own with everything else building upon it, while not addressing encoding challenges on introduction of the feature that would need to deal with it leads to half a feature that unnecessarily limits what certain ecosystems with (imo) perfectly legit use case scenarios like UTF-16 can do efficiently.
So, looking at this slide it mentions utf8 exclusively similar to what we have with WebIDL. Not quite sure how it would solve the underlying issue, that is making a compatible string from raw bytes, if it moves the problem from "directly allocating a string compatible with WebIDL bindings" to "creating a DOMString/anyref compatible with ⛄️-bindings" (if I understood this correctly?). For instance, TextEncoder doesn't support UTF-16LE (anymore), but TextDecoder does.
I'd expect that at some point in either implementation "making a compatible string from raw bytes" will be necessary anyway if the primary string implementation is provided by the module, which is likely. Please correct me if I'm missing something here. Ultimately, the issue doesn't have to be solved in the WebIDL spec, but any other spec solving it would be perfectly fine as well - as long as it is solved.
Didn't mean to sound like I was saying it was :x, sorry. I was thinking that if AssemblyScript was using UTF-16 for easier FFI with JS, then in the presence of something-bindings it would be possible to decouple that ABI. And also that AssemblyScript would probably have an easier time of making that ABI switch than a more-ossified target like .NET, on account of it's a younger platform.
All good, your point makes perfect sense. Just wanted to emphasize that, even if AssemblyScript would make this change, this is a broader problem than what it might look like from this issue alone :)
I thought if WebAssembly implement WebIDL binding it should follow WebIDL spec which support three types of strings: DOMString, ByteString and USVString. Most of WebIDL which relate to WebApi mostly using DOMString which commonly interpreted as UTF-16 encoded strings [RFC2781]. ByteString is actually ASCII and at last USVString which not require concrete encoding format. Addisianal note about USVString from WebIDL spec:
Specifications should only use USVString for APIs that perform text processing and need a string of Unicode scalar values to operate on. Most APIs that use strings should instead be using DOMString, which does not make any interpretations of the code units in the string. When in doubt, use DOMString.
UTF-8 was intentionally designed as a strict super-set of ASCII, therefore UTF-8 can be used to efficiently transfer ASCII text.
Yeah, tried to be careful there (in regards to C-strings) but the more I think about it the less I believe that this distinction is necessary, especially since any API being bound will very likely be reasonably modern anyway. So that'd leave us with UTF-8 and UTF-16LE. Anything else you could imagine would fit there in terms of "well-known encodings" (in context of modern programming languages)?
@Pauan WebIDL (except ByteString) and Javascript not using ASCII at all. Strings in javascript represented as UTF-16LE by default but v8 for example can represent strings in different ways and encodings internally. For example during concatenation strings can represent as rope structure which flattened to "normal" string before serialization / father conversion or before passing to Web Api. But that doesn't mean we should use rope structure as default structure for string for example. The same with UTF8
Side note: USVString looks like it can be described in terms of UTF-32 (not sure if that makes sense as I don't know anything using it for its internal representation). But maybe the least common denominator is UTF here?
Specifications should only use ByteString for interfacing with protocols that use bytes and strings interchangeably, such as HTTP. In general, strings should be represented with DOMString values, even if it is expected that values of the string will always be in ASCII or some 8 bit character encoding. Sequences or frozen arrays with octet or byte elements, Uint8Array, or Int8Array should be used for holding 8 bit data rather than ByteString.
JS strings will always be externref, end of story.
Is this what the community has decided, or is it a personal opinion?
The "community" does not decide things, things are decided by the WebAssembly Working Group. And yes it was decided years ago that externref is the only viable way for the host to expose things to Wasm, because of security reasons.
That is not what I read in your sentence. The tone of the comment does not seem to match the code of conduct, and so it has created many questions for me and those who have already left this community.
@aminya You asked who had made the design and decision for externref, and I answered.
The community of course is free to share their opinions, give advice, give insight, etc. but WebAssembly is not a democracy, decisions are not made by voting, decisions are not made by popular opinion, all decisions are made by consensus of the Working Group. That is how it has always worked.
Note that I am not a part of the Working Group, I have no authority to make decisions (so I am just the same as you). So please stop this "us vs them" mentality, it is not helpful.
The Working Group has decades of experience with the web. Most Working Group members are from Mozilla, Google, Microsoft, and Apple. They also work on the browsers and JS engines, so they are deeply familiar with both the web and the implementation within the browsers. They also have a lot of experience with non-JS languages and non-JS runtimes, and they are well versed in computer science academia. They are the most qualified and experienced to make decisions. You should respect them rather than assuming that they are wrong. Maybe they know things that you don't, so you should listen when they try to explain things.
It is not a violation of the code of conduct for me to answer your question and explain how the system works. Disagreement is also not a violation of the code of conduct (and it is not even morally wrong), especially when the disagreement is on technical matters.
You will not be able to get your way by trying to bully other people by falsely claiming that they are "insulting you" or "mobbing you" or "violating the code of conduct". Please do show the part of the code of conduct which you think I have violated. Quote the section of the code of conduct which I have violated, and quote my message which violates that section.
The community of course is free to share their opinions, give advice, give insight, etc. but WebAssembly is not a democracy, decisions are not made by voting, decisions are not made by popular opinion, all decisions are made by consensus of the Working Group. That is how it has always worked.
To be clear, the advancement of a proposal through all but the last phase of standardisation is based on Community Group consensus, and our process hasn't been tested yet by a divergence between Community Group and Working Group consensus (which would only matter for the final phase of standardisation). My expectation would be that such a divisive proposal would be rejected at an earlier phase, since Working Group members also contribute to the consensus of the Community Group.
The proposal here could be brought before the Community Group in an attempt to seek consensus for phase 1, and no one, Working Group member or otherwise, is going to stand in the way of this process. It would be the job of the proposal champion to convince the community to achieve consensus for each phase transition.
Without pre-supposing whether the Community Group would be able to form consensus, I'd point out that all the people involved in this conversation are part of this community, and the same debates regarding the extent to which the proposal privileges one set of string representations will likely be played out again in any meeting. These issue conversations can only ever be small-scale attempts to "feel for" how a Community Group consensus-seeking exercise would play out.
It is incredibly unreasonable and arrogant to tell them to "just rewrite your code to use this different string type".
First and foremost, nobody said what's in quotation marks there, and while laying words into another participants mouth you even have the audacity to insult them for it?
I am much more concerned however with the roles the respective CG members, including the champion of this proposal, who is a co-creator of WebAssembly, or one Google employee in another issue where I asked about GC arrays, played in the broader picture of systematic bullying for all these years. This may well be a W3C group failing to uphold minimal standards, and tolerating that for an extended period of time. In my opinion this abuse can only be resolved by decisively expelling those involved for repeatedly and intentionally violating the values the CG once pledged to uphold.
Note that I am not a part of the Working Group, I have no authority to make decisions (so I am just the same as you).
You say:
#13 (comment)
JS strings will always be externref, end of story.
Since you are clearly not willing to argue in good faith
If you are not a member of the working group, you are not eligible for specifying the end of the story, specifying if people are arguing in good faith, or even specifying that a proposal is good or not. You can just say:
"In my opinion, this proposal does not satisfy my standards".
The impression I get from your above comments is that you are the only decision-maker for the whole community.
@dcodeIO Universal strings (under your current proposal) would require languages to rewrite countless thousands of libraries to accommodate the new string type. I have explained the many technical flaws with your proposal, yet you refuse to change your universal strings proposal.
So the only conclusion I can see is that you think these costs are justified, and that languages like Python should just be ignored. In fact both you and @MaxGraey have outright stated that languages like Python are unimportant and can be ignored. That is obviously unacceptable for a universal string type, especially since Python is one of the most popular languages in the world.
Perhaps I could have phrased it better, and I apologize for that, but look at your own comments for many examples of extremely disrespectful conduct. You have repeatedly accused the Working Group of sabotaging JS, and repeatedly made many baseless claims (such as claiming that we are dismissing universal strings for no reason, even though we have given extensive technical reasons for why it will not work). You also claimed that you are being "mobbed" (even though that has never happened), and you accused the Working Group of encouraging your mobbing (even though that never happened either).
@aminya If you are not a member of the working group, you are not eligible for specifying the end of the story
That is the conclusion that the Working Group came to, I am simply explaining their conclusion. Incidentally, I agree with their conclusion, because the host and Wasm's memory cannot intermingle, therefore externref is the only logical way to support host types. You have not presented any arguments to the contrary.
specifying if people are arguing in good faith
Yes, he was not arguing in good faith. He completely ignored all of the technical points I brought up, and he kept mentioning that external strings are UTF-16 (which is completely irrelevant and changes absolutely nothing about any of my arguments). That is a logical fallacy. Whether something is a logical fallacy or not is objective, it is not a subjective opinion. I have talked extensively with @MaxGraey , and he has consistently engaged in this behavior, this is not a one-time thing.
or even specifying that a proposal is good or not. You can just say: "In my opinion, this proposal does not satisfy my standards".
The arguments I have made are technical and objective, they are not subjective opinions. None of you have engaged with any of the technical arguments I have made.
It should not be necessary to constantly repeat "in my opinion", because that should be assumed when somebody is stating their opinion.
The impression I get from your above comments is that you are the only decision-maker for the whole community.
No, I have simply repeated the conclusions that others have already made, and given objective technical explanations for why those decisions were made, and for why universal strings will not work. No decisions were made, only explanations were given. And rather than engaging with those arguments, instead you try to divert the conversation into political areas.
That will not work. Even if you are right, and I did violate the code of conduct... that still does not make your arguments correct, and it does not make universal strings correct. The only way you can prove universal strings correct is to argue on a technical level, which you have refused to do.
If you are convinced that universal strings are a good idea, then you must create a proposal for it and champion it.
So the only conclusion I can see is that you think these costs are justified, and that languages like Python should just be ignored. In fact both you and @MaxGraey have outright stated that languages like Python are unimportant and can be ignored. That is obviously unacceptable for a universal string type, especially since Python is one of the most popular languages in the world.
Are you agree compile JavaScript to wasm isn't make sense? But it can via embedding whole VM like QuickJS or JSC. And this uses by Figma's plugin for example. But it's impractical for general case. Exactly the same for Python. Btw JS has at least same popularity as Python. I just want to demonstrate to you that this is not an argument at all. There are languages that compile very well to WebAssembly, but there are also those that absolutely senselessly pull into WebAssembly. Of course, there are projects like pyodide. But these are rather exceptions to the rule. And the speed of the interop does not really matter there. also I guess I clearly explained that Python support Latin1, UTF-8, UTF16 / UCS-2 and UCS-4 seamlessly and simultaneously
@MaxGraey Btw JS has at least same popularity as Python. I just want to demonstrate to you that this is not an argument at all.
Yes... it is an argument, because the point of universal strings is to act as a universal type for interop between various Wasm languages.
It cannot fulfill that goal if it is excluding a large number of popular languages. A "universal string" type which only works for 5 languages is strictly worse than interface types, since interface types allow all languages to interop together, without giving special preference to only some languages.
There are languages that compile very well to WebAssembly, but there are also those that absolutely senselessly pull into WebAssembly.
Generally speaking that isn't true: Wasm is low level enough that almost any language can run at close to native speeds on it. And in the cases where that isn't true, new proposals are created to fix that (e.g. tail calls). Wasm tries hard to not give preferential treatment to languages or groups of languages.
In any case, that is not Wasm's place to decide. Wasm does not decide which languages are "worthy" of being on Wasm, instead it should try to treat all languages as equally as it can.
Imagine the amount of drama that would happen if Wasm started privilege certain languages over others. Or if it started excluding very popular languages like Python.
Even this very thread was started because of the UTF-8 preference in the old proposal for interface types. And yet you are trying to do the same sort of exclusion.
And the speed of the interop does not really matter there.
You do not speak on behalf of those languages, or the projects using those languages. Many of them care very much so about speed.
The words you use do matter to me and many other community members. Please read the code of conduct
That is the conclusion that the Working Group came to, I am simply explaining their conclusion.
If you want to restate someone's opinion, there are ways for that.
You should say: "The working group has concluded that JsStrings will be only externref" instead of saying "JS strings will always be externref, end of story"
Yes, he was not arguing in good faith. He completely ignored all of the technical points I brought up, and he kept mentioning that external strings are UTF-16 (which is completely irrelevant and changes absolutely nothing about any of my arguments). That is a logical fallacy. Whether something is a logical fallacy or not is objective, it is not a subjective opinion. I have talked extensively with @MaxGraey , and he has consistently engaged in this behavior, this is not a one-time thing.
I don't have the same idea, and I'm sure many others are the same, and because you are not the community manager, you are not eligible to conclude things just based on your personal opinion.
The arguments I have made are technical and objective, they are not subjective opinions. None of you have engaged with any of the technical arguments I have made.
Anything is subjective. Even the rules of physics don't hold in all the conditions.
It should not be necessary to constantly repeat "in my opinion", because that should be assumed when somebody is stating their opinion.
We can't assume things. That is not how language works. Based on the code of conduct, you should be as clear as possible.
Even if you are right, and I did violate the code of conduct.
Based on the code of conduct, you should
Acknowledge that you've done something improper
Briefly apologize. Don't try to explain yourself or minimize the issue
If possible, edit your message, restate your communication in a better way or withdraw your statement. Publicly revising your statement helps define the culture for others
Generally speaking that isn't true: Wasm is low level enough that almost any language can run at close to native speeds on it. And in the cases where that isn't true, new proposals are created to fix that (e.g. tail calls). Wasm tries hard to not give preferential treatment to languages or groups of languages.
Sorry, but you're really don't understand what is WebAssembly and how it produced from different languages
We seem to have already agreed that we will ignore each other? But you still continue to discuss with me. Why?
In fact both you and @MaxGraey have outright stated that languages like Python are unimportant and can be ignored
And yet you are doing it again. I never said that. I'd be very interested to look into the concrete implementation details, actually, and see where I may perhaps be able to help. It may turn out that this is not possible of course, but what you are saying is untrue, and sure it is supported by those encouraging you to target me with it instead of stepping in.
@aminya You should say: "The working group has concluded that JsStrings will be only externref" instead of saying "JS strings will always be externref, end of story"
I agree that I could have phrased it better, but even then I do not think that my words were against the code of conduct.
I don't have the same idea, and I'm sure many others are the same, and because you are not the community manager, and so you are not eligible to conclude things just based on your personal opinion.
You can disagree if you like, but fallacies are objective.
I do not know why you are bringing up community managers. Anybody is free to make whatever discussions or conclusions they like, and also to repeat the conclusions of others, that is not something that only community managers are allowed to do.
Any discussions in this thread are not decisions, because decisions are made by consensus of the Working Group, which is handled via face-to-face meetings.
Anything is subjective. Even the rules of physics don't hold in all the conditions.
That is blatantly false, even more so in this discussion, since computers are intentionally designed to be objective and logical, and we are discussing computer systems.
We can't assume things. That is not how language works.
Language is based almost entirely on assumptions and context, it is very fuzzy. That is how humans work, humans are not robots. You seem to have a very extreme view of the code of conduct, you are wielding it as a weapon to use against anybody you disagree with, so I encourage you to contact an Ombudsperson.
I am not going to argue this matter anymore, since it's clear that you are unwilling to engage in any technical arguments, and are simply trying to strongarm others into silence.
That's not how I think. Based on the code of conduct, you are not allowed to use that word (blantly) against anyone's opinion.
Language is based almost entirely on assumptions and context, it is very fuzzy. That is how humans work, humans are not robots.
That's not how the code of conduct is written, and you can't change the code of conduct based on your personal opinion.
I am not going to argue this matter anymore, since it's clear that you are unwilling to engage in any technical arguments, and are simply trying to strongarm others into silence.
Three years of technical arguments have not been enough, and so the people are so tired that they want to leave this community.
You seem to have a very extreme view of the code of conduct, you are wielding it as a weapon to use against anybody you disagree with, so I encourage you to contact an Ombudsperson.
I am giving you a chance to fix the issue informally.
Adding a note here that personal attacks on technical discussions are not okay. I will be locking this issue down for now as it could use a cool down period. For technical discussions that need an avenue, I suggest adding agenda items to an upcoming CG meeting.
Activity
dcodeIO commentedon Jun 30, 2019
Since this hasn't received any comments yet, allow me to bump this: I am still curious if UTF-16 strings can be supported. In AssemblyScript's case all strings are UTF-16LE already, so only having the option to re-encode (potentially twice if the bound API wants UTF-16) does seem like it should be taken into account.
fgmccabe commentedon Jul 1, 2019
dcodeIO commentedon Jul 1, 2019
To me it looks like having such an operator can lead to significantly less work where UTF-16 is already present on both sides of the equation, while implementing the APIs in any case where either side is UTF-8 can easily be implemented by reencoding conditionally. Hence, the module would chose the operator that fits its internal string layout ideally, and the host would do whatever is necessary to make it fit into theirs. This leaves us with those cases:
while avoiding the very unfortunate case of
I see, yet I thought it might make sense to raise this early so once committing to operators, this case is well thought through :)
jgravelle-google commentedon Jul 1, 2019
Yeah that would be unfortunate.
The real question is, can we skip JS entirely? At which point, what does the host API use internally? For example a similarly bad outcome would be
So I think the sanest way to handle that is with a declarative API on the bindings layer. Which is what you said earlier:
So the higher-level point is that we should be able to adequately describe the most common/reasonable ways to encode strings, so that we can minimize the number of encodings in the best case.
Agree + disagree. On the one, it's all in the sketch stage at the moment, where we're feeling out the rough edges. So from a managing-expectations point of view, this makes sense to say.
On the other hand, it's kind of incongruous to say "everything's up in the air, so don't raise any design issues." I don't think that was the intent, but that's kind of how it sounded. A more real translation of how I heard it was "don't worry about this now, we'll figure it out later." To which I would say, as a general principle, that yes we'll figure it out later, but we should raise it now to figure out if we should worry about it now. Especially because multiple people can think about different bits of the spec asynchronously.
jgravelle-google commentedon Jul 1, 2019
Also something I should mention explicitly:
I find it incredibly likely that we will default to 1 binding expression per ⛄-type per wasm representation (e.g. 1 for linear memory and 1 for gc), which is to say the MVP of ⛄-bindings will have one binding expression per type, because gc will probably not be shipped yet. On that basis, we will probably start with only UTF-8-encoding (I imagine we will drop the utf8-cstr binding too, for similar reasons).
My general mental model here is that we can always add bindings in the future as we find a need for them. And it may be the case that in practice, the re-encoding from UTF-16 isn't enough of a bottleneck to be worth it. Unless it is, at which point we can add that binding, and it will be more obviously useful because we'll have much more real-world data.
Also for AssemblyScript specifically, would it be reasonable to change the internal string representation from UTF-16 to UTF-8 in the presence of ⛄-bindings? It is, after all, "Definitely not a TypeScript to WebAssembly compiler" 😄
dcodeIO commentedon Jul 1, 2019
At the end of the day we are building just tools here and one can't know the use case of everyone. Like, any use case extensively calling bound functions with string arguments would hit this and my expectation would be that this'll happen anyway (in certain use cases). Like, if we'd wait, this'll surface sooner or later, so it can as well be addressed from the start, instead of having to tell everyone running into this that their use case is currently not well-supported even through we did see it coming. Especially since specification and implementation of new operators can take a long time again.
I'm sorry, the "⛄️-bindings" term is new to me. Would you point me into the right direction where I can learn about it? :)
Regarding UTF-8: In fact we have been thinking about this but it doesn't seem feasible, because we are re-implemeting
Stringafter the JS-API (with other stdlib components relying on it) and going with something else than UCS-2 representation seems suboptimal since the API is so deeply rooted into the language that mimicking UCS-2 would cost too much perf-wise. After all we are trying to stay as close to TS as reasonable to make picking up AssemblyScript a smooth experience. Also would like to note that this isn't exclusively an AssemblyScript thing, as other languages are using UTF-16LE as well, like everything in the .NET/Mono space.jgravelle-google commentedon Jul 1, 2019
It's ultimately a tradeoff. My thoughts here are that it will be strictly easier to spec and implement a bindings proposal that defines 8 operators, as opposed to one that defines 40. So we could just add UTF-16, but we could also just add C-strings and we could just add Scheme cons-list strings and we could just add Haskell lazy cons thunks, and so on. So for MVP I think we need to be really strict as to what exactly is "minimal", and in this context Minimal means "we can reason about strings at all".
We also need to balance the "viable" portion. Originally I was thinking we should avoid reasoning about strings and allocators at all, due to the complexity they add. Further discussion on this (see: #25) made me realize that not having an answer for allocators would compromise the viability of the proposal entirely. On that basis, not having UTF-16 support from day 1 is unlikely to leave the bindings proposal dead in the water.
By means of analogy, I would rather we ship anyref without waiting for the full gc proposal, because anyref on its own is a very enabling feature. It is in many ways suboptimal, but it it is more useful than what we had before. On that basis, I want to be very cautious about adding scope to the bindings MVP, especially when that scope is separable to a v2 that describes an expanded set of binding expressions.
Sure, @lukewagner presented at the June CG meeting, and here's the slide deck: https://docs.google.com/presentation/d/1wtAknL-UJWDoIgSbyF5paTBSpVVj-fKU4tiHMxJbSzE/edit
tl;dr does this wasm binding layer we're describing need to reason about WebIDL at its core, or is WebIDL another target with a produce/consume pair? If the latter, and we suspect that is the case, then we're free to design an IDL that better matches what we're trying to do, rather than try to retrofit that on top of WebIDL.
Full notes of the accompanying discussion here: https://github.com/WebAssembly/meetings/blob/master/2019/CG-06.md#webidl-bindings-1-2-hrs
Didn't mean to sound like I was saying it was :x, sorry. I was thinking that if AssemblyScript was using UTF-16 for easier FFI with JS, then in the presence of something-bindings it would be possible to decouple that ABI. And also that AssemblyScript would probably have an easier time of making that ABI switch than a more-ossified target like .NET, on account of it's a younger platform.
dcodeIO commentedon Jul 2, 2019
Makes sense, yeah. Though, to me it seems not overly complex to have a (potentially extensible) immediate operand on
str(/alloc-str) that indicates a well-known encoding. I'd consider UTF-8, UTF-16LE and maybe ASCII here (not sure), with length always provided by the caller (even if null-terminated), but I'm certainly not an expert in this regard.I totally agree with the
anyrefmention, but don't entirely agree on the comparison with encodings.anyrefis a useful feature on its own with everything else building upon it, while not addressing encoding challenges on introduction of the feature that would need to deal with it leads to half a feature that unnecessarily limits what certain ecosystems with (imo) perfectly legit use case scenarios like UTF-16 can do efficiently.Thanks! :)
So, looking at this slide it mentions utf8 exclusively similar to what we have with WebIDL. Not quite sure how it would solve the underlying issue, that is making a compatible string from raw bytes, if it moves the problem from "directly allocating a string compatible with WebIDL bindings" to "creating a DOMString/anyref compatible with ⛄️-bindings" (if I understood this correctly?). For instance,
TextEncoderdoesn't support UTF-16LE (anymore), butTextDecoderdoes.I'd expect that at some point in either implementation "making a compatible string from raw bytes" will be necessary anyway if the primary string implementation is provided by the module, which is likely. Please correct me if I'm missing something here. Ultimately, the issue doesn't have to be solved in the WebIDL spec, but any other spec solving it would be perfectly fine as well - as long as it is solved.
All good, your point makes perfect sense. Just wanted to emphasize that, even if AssemblyScript would make this change, this is a broader problem than what it might look like from this issue alone :)
MaxGraey commentedon Jul 2, 2019
I thought if WebAssembly implement WebIDL binding it should follow WebIDL spec which support three types of strings: DOMString, ByteString and USVString. Most of WebIDL which relate to WebApi mostly using DOMString which commonly interpreted as UTF-16 encoded strings [RFC2781]. ByteString is actually ASCII and at last USVString which not require concrete encoding format. Addisianal note about USVString from WebIDL spec:
Pauan commentedon Jul 2, 2019
UTF-8 was intentionally designed as a strict super-set of ASCII, therefore UTF-8 can be used to efficiently transfer ASCII text.
dcodeIO commentedon Jul 2, 2019
Yeah, tried to be careful there (in regards to C-strings) but the more I think about it the less I believe that this distinction is necessary, especially since any API being bound will very likely be reasonably modern anyway. So that'd leave us with UTF-8 and UTF-16LE. Anything else you could imagine would fit there in terms of "well-known encodings" (in context of modern programming languages)?
MaxGraey commentedon Jul 2, 2019
@Pauan WebIDL (except ByteString) and Javascript not using ASCII at all. Strings in javascript represented as UTF-16LE by default but v8 for example can represent strings in different ways and encodings internally. For example during concatenation strings can represent as rope structure which flattened to "normal" string before serialization / father conversion or before passing to Web Api. But that doesn't mean we should use rope structure as default structure for string for example. The same with UTF8
dcodeIO commentedon Jul 2, 2019
Side note: USVString looks like it can be described in terms of UTF-32 (not sure if that makes sense as I don't know anything using it for its internal representation). But maybe the least common denominator is UTF here?
MaxGraey commentedon Jul 2, 2019
About ByteString in WebIDL
Pauan commentedon Jul 2, 2019
@MaxGraey I am aware. The purpose of WebIDL bindings is to allow many different languages to use WebIDL APIs without using JavaScript.
Since each language does things differently, that means there needs to be a way to convert from one type to another type.
That's why there's a UTF-8 -> WebIDL string conversion, to allow for languages like Rust to use WebIDL bindings (since Rust uses UTF-8).
141 remaining items
aminya commentedon Feb 18, 2021
That is not what I read in your sentence. The tone of the comment does not seem to match the code of conduct, and so it has created many questions for me and those who have already left this community.
https://www.w3.org/Consortium/cepc/
Pauan commentedon Feb 18, 2021
@aminya You asked who had made the design and decision for
externref, and I answered.The community of course is free to share their opinions, give advice, give insight, etc. but WebAssembly is not a democracy, decisions are not made by voting, decisions are not made by popular opinion, all decisions are made by consensus of the Working Group. That is how it has always worked.
Note that I am not a part of the Working Group, I have no authority to make decisions (so I am just the same as you). So please stop this "us vs them" mentality, it is not helpful.
The Working Group has decades of experience with the web. Most Working Group members are from Mozilla, Google, Microsoft, and Apple. They also work on the browsers and JS engines, so they are deeply familiar with both the web and the implementation within the browsers. They also have a lot of experience with non-JS languages and non-JS runtimes, and they are well versed in computer science academia. They are the most qualified and experienced to make decisions. You should respect them rather than assuming that they are wrong. Maybe they know things that you don't, so you should listen when they try to explain things.
It is not a violation of the code of conduct for me to answer your question and explain how the system works. Disagreement is also not a violation of the code of conduct (and it is not even morally wrong), especially when the disagreement is on technical matters.
You will not be able to get your way by trying to bully other people by falsely claiming that they are "insulting you" or "mobbing you" or "violating the code of conduct". Please do show the part of the code of conduct which you think I have violated. Quote the section of the code of conduct which I have violated, and quote my message which violates that section.
conrad-watt commentedon Feb 18, 2021
To be clear, the advancement of a proposal through all but the last phase of standardisation is based on Community Group consensus, and our process hasn't been tested yet by a divergence between Community Group and Working Group consensus (which would only matter for the final phase of standardisation). My expectation would be that such a divisive proposal would be rejected at an earlier phase, since Working Group members also contribute to the consensus of the Community Group.
The proposal here could be brought before the Community Group in an attempt to seek consensus for phase 1, and no one, Working Group member or otherwise, is going to stand in the way of this process. It would be the job of the proposal champion to convince the community to achieve consensus for each phase transition.
Without pre-supposing whether the Community Group would be able to form consensus, I'd point out that all the people involved in this conversation are part of this community, and the same debates regarding the extent to which the proposal privileges one set of string representations will likely be played out again in any meeting. These issue conversations can only ever be small-scale attempts to "feel for" how a Community Group consensus-seeking exercise would play out.
dcodeIO commentedon Feb 18, 2021
@Pauan I can give one:
First and foremost, nobody said what's in quotation marks there, and while laying words into another participants mouth you even have the audacity to insult them for it?
I am much more concerned however with the roles the respective CG members, including the champion of this proposal, who is a co-creator of WebAssembly, or one Google employee in another issue where I asked about GC arrays, played in the broader picture of systematic bullying for all these years. This may well be a W3C group failing to uphold minimal standards, and tolerating that for an extended period of time. In my opinion this abuse can only be resolved by decisively expelling those involved for repeatedly and intentionally violating the values the CG once pledged to uphold.
aminya commentedon Feb 18, 2021
@Pauan
You say:
If you are not a member of the working group, you are not eligible for specifying the end of the story, specifying if people are arguing in good faith, or even specifying that a proposal is good or not. You can just say:
"In my opinion, this proposal does not satisfy my standards".
The impression I get from your above comments is that you are the only decision-maker for the whole community.
Pauan commentedon Feb 18, 2021
@dcodeIO Universal strings (under your current proposal) would require languages to rewrite countless thousands of libraries to accommodate the new string type. I have explained the many technical flaws with your proposal, yet you refuse to change your universal strings proposal.
So the only conclusion I can see is that you think these costs are justified, and that languages like Python should just be ignored. In fact both you and @MaxGraey have outright stated that languages like Python are unimportant and can be ignored. That is obviously unacceptable for a universal string type, especially since Python is one of the most popular languages in the world.
Perhaps I could have phrased it better, and I apologize for that, but look at your own comments for many examples of extremely disrespectful conduct. You have repeatedly accused the Working Group of sabotaging JS, and repeatedly made many baseless claims (such as claiming that we are dismissing universal strings for no reason, even though we have given extensive technical reasons for why it will not work). You also claimed that you are being "mobbed" (even though that has never happened), and you accused the Working Group of encouraging your mobbing (even though that never happened either).
That is the conclusion that the Working Group came to, I am simply explaining their conclusion. Incidentally, I agree with their conclusion, because the host and Wasm's memory cannot intermingle, therefore
externrefis the only logical way to support host types. You have not presented any arguments to the contrary.Yes, he was not arguing in good faith. He completely ignored all of the technical points I brought up, and he kept mentioning that external strings are UTF-16 (which is completely irrelevant and changes absolutely nothing about any of my arguments). That is a logical fallacy. Whether something is a logical fallacy or not is objective, it is not a subjective opinion. I have talked extensively with @MaxGraey , and he has consistently engaged in this behavior, this is not a one-time thing.
The arguments I have made are technical and objective, they are not subjective opinions. None of you have engaged with any of the technical arguments I have made.
It should not be necessary to constantly repeat "in my opinion", because that should be assumed when somebody is stating their opinion.
No, I have simply repeated the conclusions that others have already made, and given objective technical explanations for why those decisions were made, and for why universal strings will not work. No decisions were made, only explanations were given. And rather than engaging with those arguments, instead you try to divert the conversation into political areas.
That will not work. Even if you are right, and I did violate the code of conduct... that still does not make your arguments correct, and it does not make universal strings correct. The only way you can prove universal strings correct is to argue on a technical level, which you have refused to do.
If you are convinced that universal strings are a good idea, then you must create a proposal for it and champion it.
MaxGraey commentedon Feb 18, 2021
Are you agree compile JavaScript to wasm isn't make sense? But it can via embedding whole VM like QuickJS or JSC. And this uses by Figma's plugin for example. But it's impractical for general case. Exactly the same for Python. Btw JS has at least same popularity as Python. I just want to demonstrate to you that this is not an argument at all. There are languages that compile very well to WebAssembly, but there are also those that absolutely senselessly pull into WebAssembly. Of course, there are projects like pyodide. But these are rather exceptions to the rule. And the speed of the interop does not really matter there. also I guess I clearly explained that Python support Latin1, UTF-8, UTF16 / UCS-2 and UCS-4 seamlessly and simultaneously
Pauan commentedon Feb 18, 2021
Yes... it is an argument, because the point of universal strings is to act as a universal type for interop between various Wasm languages.
It cannot fulfill that goal if it is excluding a large number of popular languages. A "universal string" type which only works for 5 languages is strictly worse than interface types, since interface types allow all languages to interop together, without giving special preference to only some languages.
Generally speaking that isn't true: Wasm is low level enough that almost any language can run at close to native speeds on it. And in the cases where that isn't true, new proposals are created to fix that (e.g. tail calls). Wasm tries hard to not give preferential treatment to languages or groups of languages.
In any case, that is not Wasm's place to decide. Wasm does not decide which languages are "worthy" of being on Wasm, instead it should try to treat all languages as equally as it can.
Imagine the amount of drama that would happen if Wasm started privilege certain languages over others. Or if it started excluding very popular languages like Python.
Even this very thread was started because of the UTF-8 preference in the old proposal for interface types. And yet you are trying to do the same sort of exclusion.
You do not speak on behalf of those languages, or the projects using those languages. Many of them care very much so about speed.
aminya commentedon Feb 18, 2021
@Pauan
The words you use do matter to me and many other community members. Please read the code of conduct
If you want to restate someone's opinion, there are ways for that.
You should say: "The working group has concluded that JsStrings will be only externref" instead of saying "JS strings will always be externref, end of story"
I don't have the same idea, and I'm sure many others are the same, and because you are not the community manager, you are not eligible to conclude things just based on your personal opinion.
Anything is subjective. Even the rules of physics don't hold in all the conditions.
We can't assume things. That is not how language works. Based on the code of conduct, you should be as clear as possible.
Based on the code of conduct, you should
https://www.w3.org/Consortium/cepc/#mistake
If you don't intend to do that, I have no choice but to raise the issue directly with the Ombudspeople as a group or individually.
MaxGraey commentedon Feb 18, 2021
Sorry, but you're really don't understand what is WebAssembly and how it produced from different languages
We seem to have already agreed that we will ignore each other? But you still continue to discuss with me. Why?
dcodeIO commentedon Feb 18, 2021
And yet you are doing it again. I never said that. I'd be very interested to look into the concrete implementation details, actually, and see where I may perhaps be able to help. It may turn out that this is not possible of course, but what you are saying is untrue, and sure it is supported by those encouraging you to target me with it instead of stepping in.
Pauan commentedon Feb 18, 2021
I agree that I could have phrased it better, but even then I do not think that my words were against the code of conduct.
You can disagree if you like, but fallacies are objective.
I do not know why you are bringing up community managers. Anybody is free to make whatever discussions or conclusions they like, and also to repeat the conclusions of others, that is not something that only community managers are allowed to do.
Any discussions in this thread are not decisions, because decisions are made by consensus of the Working Group, which is handled via face-to-face meetings.
That is blatantly false, even more so in this discussion, since computers are intentionally designed to be objective and logical, and we are discussing computer systems.
Language is based almost entirely on assumptions and context, it is very fuzzy. That is how humans work, humans are not robots. You seem to have a very extreme view of the code of conduct, you are wielding it as a weapon to use against anybody you disagree with, so I encourage you to contact an Ombudsperson.
I am not going to argue this matter anymore, since it's clear that you are unwilling to engage in any technical arguments, and are simply trying to strongarm others into silence.
aminya commentedon Feb 18, 2021
That's not how I think. Based on the code of conduct, you are not allowed to use that word (blantly) against anyone's opinion.
That's not how the code of conduct is written, and you can't change the code of conduct based on your personal opinion.
Three years of technical arguments have not been enough, and so the people are so tired that they want to leave this community.
I am giving you a chance to fix the issue informally.
Based on section 5 of the code of conduct:
"If you don't understand what you did wrong, assume that the hurt party has good cause and accept it."
dtig commentedon Feb 18, 2021
Adding a note here that personal attacks on technical discussions are not okay. I will be locking this issue down for now as it could use a cool down period. For technical discussions that need an avenue, I suggest adding agenda items to an upcoming CG meeting.