Repository navigation
The Road to Stabilizing the Standard Library #4600
Description
The Standard Library (std) aims to provide high-quality, reliable APIs for fearless use in JavaScript and TypeScript projects. For some time, how and when std would reach stability has been in question. Now, thanks to JSR and workspaces functionality in the Deno runtime, we have a path forward.
What stabilization will look like
Currently, std acts as a single module with a single version, meaning a user may be inclined to upgrade, even if a release doesn't affect the sub-module they use.
After stabilization, new std packages will be exclusively hosted on JSR as separate packages with independent versions. Deno's workspaces functionality allows us to treat each top-level subdirectory as its own package and better adhere to Semantic Versioning. Dependencies become more manageable by only requiring an upgrade for the packages you use, not the entire std module.
More importantly, independent versioning allows us to work on stabilizing packages asynchronously. We can stabilize more established packages while preparing other packages for stabilization. In other words, we can progressively deliver stable packages for the community sooner rather than later.
These changes will also mean we will release std packages independently of Deno runtime releases.
How we'll achieve stability
Step 1 - Decouple std from the runtime (complete)
deno and deno_std are now effectively decoupled, so releasing them in tandem is not required. The remaining point of friction is apiland_scripts, which generates docs. We now have two separate GH Actions in apiland_scripts that can be run independently (release CLI and release STD). These actions can be run independently, and it's a matter of deciding if we do the "split".
Step 2 - Publish exclusively to JSR (complete)
At the moment, std is published to deno.land/std as-is, then converted into a JSR-compatible form in CI and published to JSR as separate packages under the @std scope. Once the previous step is complete, the codebase can be permanently converted into its JSR-compatible form, providing the granular package versioning for the next step.
Note: For the foreseeable future, deno.land/std will remain online and accessible until the version before the JSR cutover is complete.
Step 3 - Stabilize packages asynchronously (in progress)
Once std is entirely cut over to JSR and packages become more stable, we can begin stabilizing packages. A package will only be stabilized once strict criteria are met that guarantee its quality. These will require that each package is:
- Reviewed and approved with sufficient consensus
- Tested for maximum coverage
- Fully documented and well-designed
- Void of any ongoing issues or known bugs
- Well supported across many versions of the Deno runtime
Each package follows the stabilization process defined in #4661
Current work
Work towards stabilizing each package is already in progress. This work includes:
- Completing documentation (tracking: complete documentation #3764)
- Increasing test coverage (tracking: improve test coverage #3713)
- Fixing bugs and adding missing features
- Closing issues and PRs
- Improving API design
You can track the stability status of each package here:
| Package | Docs | Test | RC | 1.0.0 | The issue | Stabilization Date |
|---|---|---|---|---|---|---|
| bytes | ✅ | ✅ | ✅ | ✅ | #4629 | Jun 6, 2024 |
| collections | ✅ | ✅ | ✅ | ✅ | #4647 | Jun 20, 2024 |
| media-types | ✅ | ✅ | ✅ | ✅ | #4730 | Jun 24, 2024 |
| crypto | ✅ | ✅ | ✅ | ✅ | #4885 | Jun 30, 2024 |
| encoding | ✅ | ✅ | ✅ | ✅ | #4856 | Jun 30, 2024 |
| uuid | ✅ | ✅ | ✅ | ✅ | #4748 | Jun 30, 2024 |
| assert | ✅ | ✅ | ✅ | ✅ | #4717 | Jul 6, 2024 |
| data-structures | ✅ | ✅ | ✅ | ✅ | #4986 | Jul 12, 2024 |
| html | ✅ | ✅ | ✅ | ✅ | #4921 | Jul 12, 2024 |
| msgpack | ✅ | ✅ | ✅ | ✅ | #5000 | Jul 12, 2024 |
| path | ✅ | ✅ | ✅ | ✅ | #4922 | Jul 12, 2024 |
| regexp | ✅ | ✅ | ✅ | ✅ | #4998 | Jul 12, 2024 |
| toml | ✅ | ✅ | ✅ | ✅ | #4923 | Jul 12, 2024 |
| async | ✅ | ✅ | ✅ | ✅ | #5001 | Jul 19, 2024 |
| cli | ✅ | ✅ | ✅ | ✅ | #5002 | Jul 19, 2024 |
| text | ✅ | ✅ | ✅ | ✅ | #4999 | Jul 19, 2024 |
| ulid | ✅ | ✅ | ✅ | ✅ | #4996 | Jul 19, 2024 |
| expect | ✅ | ✅ | ✅ | ✅ | #5014 | Jul 26, 2024 |
| front-matter | ✅ | ✅ | ✅ | ✅ | #5016 | Jul 26, 2024 |
| fs | ✅ | ✅ | ✅ | ✅ | #5008 | Jul 26, 2024 |
| json | ✅ | ✅ | ✅ | ✅ | #5009 | Jul 26, 2024 |
| jsonc | ✅ | ✅ | ✅ | ✅ | #5011 | Jul 26, 2024 |
| streams | ✅ | ✅ | ✅ | ✅ | #5004 | Jul 26, 2024 |
| yaml | ✅ | ✅ | ✅ | #5015 | Jul 26, 2024 | |
| fmt | ✅ | ✅ | ✅ | ✅ | #5005 | Aug 2, 2024 |
| http | ✅ | ✅ | ✅ | ✅ | #5007 | Aug 2, 2024 |
| net | ✅ | ✅ | ✅ | ✅ | #5006 | Aug 2, 2024 |
| semver | ✅ | ✅ | ✅ | ✅ | #5013 | Aug 2, 2024 |
| testing | ✅ | ✅ | ✅ | ✅ | #5010 | Aug 2, 2024 |
| csv | ✅ | ✅ | ✅ | ✅ | #5012 | Aug 5, 2024 |
Excluded Packages
| Package | Note |
|---|---|
| archive | This package is excluded because it’s in the process of a re-design. Tracked in #1658. |
| datetime | This package will be re-designed after Temporal is shipped. |
| dotenv | This package is excluded because this feature conflicts with Deno CLI --env option. |
| ini | This package is excluded because there still remain various concerns about parse and stringify behavior. See #5614 |
| io | No consensus on what should/shouldn't be included in this module yet. Ideally this module shouldn't exist. See #5059 for more details. |
| log | The design of this package is not ready yet. |
| url | This package needs more discussion. See #4992 for details. |
| webgpu | This package is excluded because WebGPU is not yet stable. Stabilization tracking issue is #5633. |
If you'd like to contribute to stabilizing the Standard Library in any of this work, please feel free to let us know, either on GitHub or Discord. All contributions are welcome and very much appreciated.
Stabilization process
Each package must go through the following steps to achieve stabilization:
- Publish version 1.0.0-rc.1 once meeting the following requirements:
- Approved by at least 2 maintainers. There must be a consensus that the design, documentation, and implementation of the package are good and that it is unlikely to undergo breaking changes in the future.
- 100% documented, passing
deno doc --lintchecks and adhering to the documentation guidelines. - Maximum reasonable test coverage
- There are no open issues or pull requests that might lead to breaking changes. For example, issues that suggest new non-breaking features are fine to exist at stabilization.
- Allow 3 weeks for the community and the core team to review the package and handle any feedback. There must be a consensus that the design, documentation, and implementation of the package are good and it is unlikely to undergo breaking changes in the future.
- If there are no remaining issues, publish version 1.0.0. If there are remaining issues, extend the waiting period to allow further time for a resolution. Then, repeat this step.
Reference: The dependency graph
The diagram below describes the dependencies between packages. The bottom packages (more dependent packages) usually need to be stabilized earlier than the top packages (more dependent packages).
Activity
jollytoad commentedon Apr 17, 2024
I'd love to see this become the gold standard of JSR packaging, esp multi-package libraries, and ideally see all scripts/tooling/cross-runtime testing used by the lib become a first class set of modules within the library itself, so that other authors can model their libraries on
@std, and benefit from its tooling. I want you follow your model closely in my own library and would be more than happy to contribute to generic tooling and@stditself.iuioiua commentedon Apr 17, 2024
That'd be great! Improving test coverage is one of the valuable contributions that can be made right now. If you're willing and able, please let us know which package(s) you'd like to look at in #3713. I'll keep you in mind when tooling is needed.
KnorpelSenf commentedon May 12, 2024
I agree that it makes sense to stabilise the different parts of the standard library at different points in time. However, it is rather annoying to have independent versions for each module.
Currently, we're moving towards a myriad of independent modules, and it adds a fair bit of accidental complexity to let the combinations of (#modules * #versions) explode like that.
I believe that it is a good idea to version the modules independently for the transitional period until almost everything is stable. After that, it would be best to sync up the versions again so that we can return to having The Standard Library again.
iuioiua commentedon May 12, 2024
If all packages share a common, synced version, how should the versions of other packages be affected if a single package needs a release?
I'm unsure what you mean here. Can you please clarify?
KnorpelSenf commentedon May 13, 2024
If there's only a single version, then this version needs to be incremented for every single package. This is fine because a stable standard library should not require a lot of fast fixes, and it is alright if the features are released at a much slower pace. (1-4 times a year maybe?)
The JavaScript ecosystem is missing a set of useful APIs that are reliable, trustworthy, omnipresent, simple, mature. The modules in this project could fill this exact gap. If they're not aiming to reinvent lodash-but-Deno, but rather trying to be sort of a gold standard (that's what I was trying to imply by calling it “The Standard Library”) then it is important to perceive this as a unit.
So basically, the effect of having independent versions is that std is now scattered into a million pieces, and there's no longer “the thing.”
jollytoad commentedon May 13, 2024
IMHO, the point is that it's a "Library", in this case, a loose collection of packages, which in turn is a collection of modules, it isn't one "thing", ie. it's not "The Standard Package".
Whilst it may seem superficially nice to align the version of each of these packages, the versions will lose all semantic meaning if kept abitrarily in sync. One package version shouldn't be bumped to line up with another if that package hasn't had similar semantic changes, it's just misleading.
For example, in a hypothetical situtation, if I'm working on a pure CLI program I may not care that
@std/httphas changes, and so unnecessary version bumps to@std/cliwould just be annoying.What I think we definitely need though is for JSR to show the latest versions for each package within a scope... jsr-io/jsr#421
KnorpelSenf commentedon May 13, 2024
I agree, and that's why I was saying that if versions are to be synced up, then the frequent releases have to stop. If you always only get a new standard lib twice a year, then you don't really run into the above problem. The first reason for this is that most packages will have changes of some sort, and the other reason is that even if there's a new release without changes to the parts you use, then this happens so rarely that it doesn't matter much.
The current way of having to do bookkeeping of a large number of modules, each with its own version and stability status, is really annoying, too.
lmxrd commentedon May 13, 2024
I agree with that having the version for the packages in STD in sync would be better for a coherent library, however I also see the argument with it not bringing any benefit in bumping a version without any changes. I am thus not sure if strictly following semantic versioning is the best option for the standard library.
I could see an alternative versioning scheme instead that could potentially work better:
patch versions would not introduce breaking changes, but minor versions could.
When using packages from the STD, it could be recommended to use
~for the version range, and manually bump minor versions when needed.kt3k commentedon May 14, 2024
Yes, we now recommend the users should use version range dependency like
@oak/oak@^16.0.0or@std/path@^0.224.0. These module specifiers will automatically get fixes and new features, if you run the program with--reloadoption.Also deno now has
deno add <mod>command. This command automatically looks up the latest version of the given jsr package, and add them todeno.jsonwith range dependency set up appropriately. So you even don't need to manage the version number by yourself.We rather recommend caret
^range, instead of tilde~. The caret^range includesminorandpatchupgrades for >1 versions, and it includespatchupgrades for0.xversions. This means you'll get fixes and new features automatically when the upstream is upgraded.kt3k commentedon May 14, 2024
@halvardssm
This sounds like a significant deviation from semver spec (semver explicitly prohibits the breaking changes in minor upgrades https://semver.org/#summary). I think we should rather follow the semver instead of introducing our own convention.
KnorpelSenf commentedon May 14, 2024
Maybe not when adding a new module for the first time, but:
^did this automatically beforedeno update-all-my-jsr-dependencies-to-latestsubcommand so people will indeed have to look up the versions manually, or employ some sort of script or other form of automation (renovate etc) that does thisBlackAsLight commentedon May 15, 2024
Someone did make a script on JSR to update their dependencies. I don't remember what it was called though.
A
deno dependency-updatewould be nice. Maybe someone should suggest that in the Deno repo.Peoples expectations isn't a reason to do something, and everyone else doing something one way doesn't mean Deno should as well. Tradition for the sake of tradition is a bad reason.
KnorpelSenf commentedon May 15, 2024
There will be a plethora of such scripts if we go ahead with this. Even Node tooling has evolved to update Deno deps: https://www.npmjs.com/package/npm-check-updates
It would be nice, but for other reasons. The point here is “look, the current situation is so complex that we even need new tooling, this is done needlessly” and I hope your takeaway from this isn't “wohoo, we created a superfluous problem, let's solve it by building new tooling!”
In fact, building intuitive stuff is a pretty good goal. The opposite of this would be to make things obscure and unexpected, and that surely isn't what we want.
You seem to have misunderstood. This is completely unrelated to tradition. The point was not “it's always been this way, let's keep it that way” (then we'd all be using Node) but rather that the rest of the industry isn't stupid. Whenever people innovate a standard library, they conclude that having a single version is good. Doing things differently just for the sake of doing them differently isn't a good approach, either. Instead, we should take a look at why everybody else does it a certain way, and not neglect their findings just because it could be “tradition.”
If we find out that they're indeed all stupid and that having 37+ independent versions is much more simple and intuitive than having 1 version, then sure, we can go on like this :)
kt3k commentedon May 21, 2024
@KnorpelSenf
I think we are still open to the idea of single versioned standard library in some form (such as #4327 . Let's keep this discussion there), but right now we need to split it into pieces to make the stabilization process going forward in a reasonable manner. (See also https://deno.com/blog/std-on-jsr#independent-versioning-for-each-standard-library-package )
Also the Deno community still keeps requesting the breaking changes to the standard library constantly. If we stabilize the entire standard library with a single version, then we probably stop accepting any of such request because the impact of such change would be too large. I personally don't think that situation meets the expectation of the Deno community.
KnorpelSenf commentedon May 21, 2024
@kt3k awesome, I wasn't aware of #4327, thanks for the reference.
I completely agree that having independent versions is the most reasonable approach as long as things are not stable yet. I tried to make this clear in my initial post already, no disagreement there :)
rojvv commentedon May 21, 2024
I’m also completely agreeing with @kt3k. I’m just worried if not all of the Deno team agrees with that: #4327 (comment)
iuioiua commentedon Aug 5, 2024
This work is now complete. PTAL at "Excluded Packages" for continued stabilization work that was not part of the main phase of stabilization. Thank you to all who helped!