Repository navigation
Regex-validated string types (feedback reset) #41160
Description
Activity
- addedAwaiting More FeedbackThis means we'd like to hear from more people who would be helped by this featureThis means we'd like to hear from more people who would be helped by this featureSuggestionAn idea for TypeScriptAn idea for TypeScript
on Oct 19, 2020 Use case 1, URL path building libraries,
/*snip*/ createTestCard : f.route() .append("/platform") .appendParam(s.platform.platformId, /\d+/) .append("/stripe") .append("/test-card") /*snip*/
These are the constraints for
.append(),- ✔️ Must start with leading forward slash (/)
- ❌ Must not end with trailing forward slash (/)
- ❌ Must not contain colon character (:); it is reserved for parameters
- ❌ Must not contain two, or more, forward slashes consecutively (//)
Use case 2,
- ❌ Hexadecimal/binary/decimal/etc. strings of non-trivial length (explosion of union types)
Use case 3, safer
RegExpconstructor (and similar functions?),new(pattern: string, flags?: PatternOf</^[gimsuy]*$/>): RegExp
- ❌
flagsshould only contain the charactersg,i,m,s,u,y - ❌ Each character should only be used once (To be fair, this condition would be hard for regexes, too, requiring negative lookahead or many states)
- ❌ Characters can be specified in any order
Reacted by Steven, IBRAGIM (PARVIZ BEKLIEV), Newbyte, Eric Rushing, Isabella Skořepová, Fabio Spampinato, Karl Tarvas, Nick Dugger, Alexander, Mulverine and 32 moreTemplate string type can only be used in conditional type, so it's really a "type validator", not a "type" itself. It also focuses more on manipulating strings, I think it's a different design goal from Regex-validated types.
It's doable to use conditional types to constrain parameters, for example taken from #6579 (comment)
declare function takesOnlyHex<StrT extends string> ( hexString : Accepts<HexStringLen6, StrT> extends true ? StrT : {__err : `${StrT} is not a hex-string of length 6`} ) : void;
However I think this parttern has several issues:
- It's not a common pattern, and cumbersome to repeat every time.
- The type parameter should be inferred, but was used in a condition before it "can" be inferred, which is unintuitive.
- TypeScript still doesn't support partial generic inferrence (Implement partial type argument inference using the _ sigil #26349) so it may be hard to use this pattern with more generic parameters.
Would this allow me to define type constraints for String to match the XML specification's Name constructs (short summary) and QNames by expressing them as regular expressions? If so, I am all for it :-)
AnyhowStep It isn't the cleanest, but with conditional types now allowing recursion, it seems we can accomplish these cases with template literal types: playground link
We can have compile-time regular expressions now.
But anything requiring conditional types and a generic type param to check is a non-feature to me.(Well, non-feature when I'm trying to use TypeScript for work. All personal projects have
--noEmitenabled because real TS programmers execute in compile-time)Reacted by OM and Antoine ViallonOpen question: For people who had upvoted #6579, what use cases still need addressing?
We have a strongly-typed filesystem library, where the user is expected to manipulate "clean types" like
FilenameorPortablePathversus literal strings (they currently obtain those types by using theasoperator on literals, or calling a validator for user-provided strings):export interface PathUtils { cwd(): PortablePath; normalize(p: PortablePath): PortablePath; join(...paths: Array<PortablePath | Filename>): PortablePath; resolve(...pathSegments: Array<PortablePath | Filename>): PortablePath; isAbsolute(path: PortablePath): boolean; relative(from: PortablePath, to: PortablePath): P; dirname(p: PortablePath): PortablePath; basename(p: PortablePath, ext?: string): Filename; extname(p: PortablePath): string; readonly sep: PortablePath; readonly delimiter: string; parse(pathString: PortablePath): ParsedPath<PortablePath>; format(pathObject: FormatInputPathObject<PortablePath>): PortablePath; contains(from: PortablePath, to: PortablePath): PortablePath | null; }
I'm investigating template literals to remove the
assyntax, but I'm not sure we'll be able to use them after all:- They don't raise errors very well
- Interfaces are a pain to type (both declaration and implementation would have to be generics)
- More generally, we would have to migrate all our existing functions to become generics, and our users would have too
The overhead sounds overwhelming, and makes it likely that there are side effects that would cause problems down the road - causing further pain if we need to revert. Ideally, the solution we're looking for would leave the code above intact, we'd just declare
PortablePathdifferently.RyanCavanaugh commented
on Dec 14, 2020 MemberAuthorMore actionsMaël Nison (@arcanis) it really sounds like you want nominal types (#202), since even if regex types existed, you'd still want the library consumer to go through the validator functions?
I have a strong use case for Regex-validated string types. AWS Lambda function names have a maximum length of 64 characters. This can be manually checked in a character counter but it's unnecessarily cumbersome given that the function name is usually composed with identifying substrings.
As an example, this function name can be partially composed with the new work done in 4.1/4.2. However there is no way to easily create a compiler error in TypeScript since the below function name will be longer than 64 characters.
type LambdaServicePrefix = 'my-application-service'; type LambdaFunctionIdentifier = 'dark-matter-upgrader-super-duper-test-function'; type LambdaFunctionName = `${LambdaServicePrefix}-${LambdaFunctionIdentifier}`; const lambdaFunctionName: LambdaFunctionName = 'my-application-service-dark-matter-upgrader-super-duper-test-function';
This StackOverflow Post I created was asking this very same question.
With the continued rise of TypeScript in back-end related code, statically defined data would be a likely strong use case for validating the string length or the format of the string.
Reacted by Edaz, Kyle a.k.a. TechSquidTV, David Doyle, Tukajo, Griffork, John Vandivier, Anton, Nicholas Morrissey, Shakil Hossain, Tomáš Hübelbauer and 2 moreReacted by Joe Calzaretta and ShinigamiTypeScript supports literal types, template literal types, and enums. I think a string pattern type is a natural extension that allows for non-finite value restrictions to be expressed.
I'm writing type definitions for an existing codebase. Many arguments and properties accept strings of a specific format:
- ❌ Formatted representation of a date, eg
"2021-04-29T12:34:56" - ❌ Comma-separated list of integers, eg
"1,2,3,4,5000" - ❌ Valid MIME type, eg
"image/jpeg" - ❌ Valid hex colour code, already mentioned several times
- ❌ Valid IPv4 or IPv6 address
Reacted by Steven, Sʜɪᴍᴜʀᴀ Yū, Fredrik Nicol, Shinigami, Hannes Widrig, undefined, Steffen Wilking, Tylor Steinberger, Jasper Dunn, Tukajo and 35 more- ❌ Formatted representation of a date, eg
fabiospampinato commented
on May 4, 2021 More actionsI'd like to argue against Ryan Cavanaugh (@RyanCavanaugh)'s claim in the first post saying that:
a large number of those use cases have been addressed, but possibly some still remain.
As it stands presently TypeScript can't even work with the following type literal:
type Digit = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9; type Just5Digits = `${Digit}${Digit}${Digit}${Digit}${Digit}`;
Throwing an "Expression produces a union type that is too complex to represent.(2590)" error.
That's the equivalent of the following regex:
/^\d{5}$/Just 5 digits in a row.
Almost all useful regexes are more complicated than that, and TypeScript already gives up with that, hence I'd argue the opposite of that claim is true: a small number of use cases have been addressed and the progress with template literals has been mostly orthogonal really.
Reacted by undefined, Hannes Widrig, Steven, Shinigami, Fredrik Nicol, Jo, Chiri Vulpes, Steffen Wilking, Jasper Dunn, Matt and 110 moreReacted by Leshnevskyi, John Vandivier, Gal Elmalah, Ahmed Raef and Elad ShamaiReacted by Jesper EngbergI just want to add to the need for this because template literals do not behave the way we think explicitly -
type UnionType = { kind: `kind_${string}`, one: boolean; } | { kind: `kind_${string}_again`, two: string; } const union: UnionType = { // ~~~~~ > Error here - /** Type '{ kind: "type1_123"; }' is not assignable to type 'UnionType'. Property 'two' is missing in type '{ kind: "type1_123"; }' but required in type '{ kind: `type1_${string}_again`; two: string; }'.ts(2322) */ kind: 'type1_123', }
this shows template literals are not unique and one can be a subset of another while that is not the intention of use. Regex would let us have a
$at the end to denote end of string that would help discriminate between the constituent types of this union clearly.Reacted by Tukajo, Jonas-Seiler-Wave, wbt and Matthew Dean88 remaining items
Nearly every use-case mentioned can already be implemented via built-in string template matching & extraction (see, for example, the "wouter" routing library for how they validate and extract route parameters from paths using this method).
The only problem is that the solution in all of these cases requires something like:
type ParseSomething<T extends string> = T extends `...${infer Something}...` ? T : never function validateSomething<T extends string>(input: ParseSomething<T>) { // input has now been validated, but we required this useless function to do it. // also very painful to create "arrays of valid somethings" }
What we really want to be able to do is throw away the
Tparameter after validation, as this would allow us to build data structures that don't care which hex code (for example), only that the hex code is valid.We want to be able to say (for example):
type HexCode = <exists S extends string> ParseHex<S> const hexCodes: HexCode[] = ['000000', 'FFFFFF'] // etc.
So, if I'm not mistaken, it seems like this issue can mostly be reduced to the introduction of existential types issue. Please upvote that one!
(The length-specific use-cases would probably also do well to upvote this length-specific issue.)
Hans Brende (@HansBrende) how would you use existential types for the below use case?
type Word = /^w+$/
type WordChar = 'A'|'B'|'C'| ... |'Z'|'a'|'b'|'c'| ... |'z'|'0'|'1'|'2'|'3'|'4'|'5'|'6'|'7'|'8'|'9'|'_' type IsWord<S extends string> = S extends `${WordChar}${infer R}` ? R extends '' ? unknown : IsWord<R> : never type Word = <exists S extends string> IsWord<S> & S
(Note:
WordCharimplementation would become a whole lot less verbose if the second issue I mentioned gets accepted, as then we could just do:type WordChar = 'ABC...Zabc...z0123456789_'[number].)Reacted by Sander Altmantype WordChar = 'A'|'B'|'C'| ... |'Z'|'a'|'b'|'c'| ... |'z'|'0'|'1'|'2'|'3'|'4'|'5'|'6'|'7'|'8'|'9'|'_'A) This doesn’t account for Unicode characters in a concise way
B) This is likely too complex for the type parser, no? I’ve had that issue when trying to write just a hex color type, which is only 9 characters at most, let alone an indefinite stringReacted by Josh DuffThis doesn’t account for Unicode characters in a concise way
True, but the original regexp doesn't either since it did not include the
iandu/vflags.
If we wanted/^\w+$/iuinstead, then we'd need to add 2 extra characters toWordChar:ſandK.Unicode-aware expressions in general would obviously be a bit more complicated, but still potentially doable in many cases, including this one (but in many cases not). That's why I stated existential types would cover most of the use-cases, but not all.
This is likely too complex for the type parser, no?
Nope. Try it and see! The compiler can handle something like 10K types in a union (not sure what the exact number is), but this is only 26 + 26 + 10 + 1 = 63 types. Your Hex color type was probably defined differently, something like
${Hex}${Hex}${Hex}${Hex}${Hex}${Hex}. That is a union of 16^6 = 16777216 possibilities--much larger. You must use recursion for these sorts of validations to work (and the typescript compiler implements tail-recursion elimination so it is very efficient).If both of the issues I linked to (existential types and string literal length--again, please upvote) were implemented, then you could implement
HexColorvery simply as follows:type HexDigit = `${0|1|2|3|4|5|6|7|8|9}`|'a'|'b'|'c'|'d'|'e'|'f' type IsHexString<S extends string> = S extends '' ? unknown : S extends `${HexDigit}${infer R}` ? IsHexString<R> : never type IsHexColor<S extends string> = S extends `#${infer R}` & {length: 7 | 9} ? IsHexString<R> : never // TA-DA! type HexColor = <exists S extends string> IsHexColor<S> & S
Reacted by Allan Oricil and Antoine ViallonThis would extremely useful for completely ditching ORMs and just using pure typed-SQL queries. Most devs use ORMs for the type-safety but it introduces a ton of method-chaining overhead then you end writing SQL anyways, just with methods.
Reacted by goosewobbler, Gabriele Tomberli and Daniel BayleyRyanCavanaugh commented
on Dec 13, 2024 MemberAuthorMore actionsI'm not clear on how regex is useful for SQL queries; can you clarify?
(One of the primary motivating use cases for tagged template literals was for being able to construct a context-aware DSL, including SQL queries and regular expressions)
I can see both sides here:
- Parsing actual RegExp types would add a significant amount of complexity to the TS language and compiler, so MS wants to avoid that as well as in any way possible
- Devs are somewhat satisfied, yet not entirely
- Current solutions cover many use cases, not some, they cannot cover
- Of those, who are covered, some are possible, yet have terrible UX (meaning, it's a workaround at best)—let's showcase that by implementing
UUIDtype:
Large table
Solution
Code
Pros
Cons
Okay, let's have a repeated type,
using string literals
(Source: https://overflow.freedit.eu/questions/68724603/how-to-create-a-uuid-template-literal-type-in-typescript)type Numeric = '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' type Alphabetic = 'a' | 'b' | 'c' | 'd' | 'e' | 'f' | 'g' | 'h' | 'i' | 'j' | 'k' | 'l' | 'm' | 'n' | 'o' | 'p' | 'q' | 'r' | 's' | 't' | 'u' | 'v' | 'w' | 'x' | 'y' | 'z' type Alphanumeric = Alphabetic | Numerictype Repeat<
Char extends string,
Count extends number,
Joined extends string = ``,
Acc extends 0[] = []
> = Acc['length'] extends Count ? Joined : Repeat<Char, Count,${Joined}${Char}, [0,...Acc]>type UUIDV4 =${Repeat<Alphanumeric, 8>}-${Repeat<Alphanumeric, 4>}-${Repeat<Alphanumeric, 4>}-${Repeat<Alphanumeric, 4>}-${Repeat<Alphanumeric, 12>}✅ Human-readable
✅ checked at compile time (in theory, see cons)❌ compiler error (too complex) Well, then just use a branded string with type guards instead ¯\_(ツ)_/¯
(Source: https://www.webdevtutor.net/blog/typescript-define-uuid-type)type UUID = string & { __uuidBrand: never };function isUUID(uuid: string): uuid is UUID {
return /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(uuid);
}✅ compiles
✅ type is narrowed down❌ checked at runtime
❌ no real meaning at compile timeOkay, last try: recursion
(Source: https://ybogomolov.me/type-level-uuid)type VersionChar = | '1' | '2' | '3' | '4' | '5';type Char =
| '0' | '1' | '2' | '3'
| '4' | '5' | '6' | '7'
| '8' | '9' | 'a' | 'b'
| 'c' | 'd' | 'e' | 'f';type Prev =
[never, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, ...never[]][X];type HasLength<S extends string, Len extends number> = [Len] extends [0]
? (S extends '' ? true : never)
: (S extends${infer C}${infer Rest}
? (Lowercase extends Char ? HasLength<Rest, Prev> : never)
: never);type Char4<S extends string> = true extends HasLength<S, 4> ? S : never;
type Char8<S extends string> = true extends HasLength<S, 8> ? S : never;
type Char12<S extends string> = true extends HasLength<S, 12> ? S : never;type VersionGroup<S extends string> = S extends
${infer Version}${infer Rest}
? (Version extends VersionChar
? (true extends HasLength<Rest, 3> ? S : never)
: never)
: never;type NilUUID = '00000000-0000-0000-0000-000000000000';
type UUID<S extends string> = S extends NilUUID
? S
: (S extends${infer S8}-${infer S4_1}-${infer S4_2}-${infer S4_3}-${infer S12}
? (S8 extends Char8
? (S4_1 extends Char4<S4_1>
? (S4_2 extends VersionGroup<S4_2>
? (S4_3 extends Char4<S4_3>
? (S12 extends Char12
? S
: never)
: never)
: never)
: never)
: never)
: never);
✅ checked at compile time ❌ quite a mouth full → terrible DX Together with the constant string length of #34692 mentioned by Hans Brende (@HansBrende) and built-in types like utility intrinsic string manipulation types (
Uppercase,Lowercase,CapitalizeandUncapitalize), that look like common RegExp operation shortcuts (similar to what has been suggested in #34692 (comment)), which actually are just syntactic sugar for the last example above (which could then be optimized under the hood), this would look so much more clean and would be way less cumbersome to implement:type UuidSegmentQuartett = `{(Alpha|number[1]){4}}` type UUID = `{UuidSegmentQuartett[2]}-{UuidSegmentQuartett}-{UuidSegmentQuartett}-{UuidSegmentQuartett}-{UuidSegmentQuartett[3]}`
Or, strings become generics (strings without type default to
string<any>) similar to arraystype UuidSegmentQuartett = `{(string<Alpha>|number[1]){4}}` // or `string<AlphaNum>[4]` // or even just `string<Hex>[4]` type UUID = `{UuidSegmentQuartett[2]}-{UuidSegmentQuartett}-{UuidSegmentQuartett}-{UuidSegmentQuartett}-{UuidSegmentQuartett[3]}`
Sebastian Hädrich (@shaedrich) Note: with existential types and string literal length (please upvote both of these issues), all the compiler problems in your first example go away:
type IsRepeated<Char extends string, S extends string> = S extends '' ? unknown : S extends `${Char}${infer R}` ? IsRepeated<Char, R> : never; // Note that here, an *existential type parameter* replaces the need for a large union: type Repeat<Char> = <exists S extends string> IsRepeated<Char, S> & S type HexDigit = `${0|1|2|3|4|5|6|7|8|9}`|'a'|'b'|'c'|'d'|'e'|'f' type XXXX = Repeat<HexDigit> & {length: 4} type XXXXXXXX = Repeat<HexDigit> & {length: 8} type XXXXXXXXXXXX = Repeat<HexDigit> & {length: 12} // Should work fine now since large union has been replaced with existential types: type UUID = `${XXXXXXXX}-${XXXX}-${XXXX}-${XXXX}-${XXXXXXXXXXXX}`
Reacted by goosewobbler, Sebastian Hädrich, Allan Oricil and Antoine ViallonI forgot to mention that my example is a little oversimplified, since
- the first character of the third pair (the version number) can only have the values 0 – 8
- there are version-specific constraints of what values other places within the rest of the UUID can take (especially the NIL and MAX UUIDs)
RyanCavanaugh commented
on Dec 16, 2024 MemberAuthorMore actionsBy far the best feature for UUID is #43335
But again, how do you even get a malformed UUID in the first place? You can only ever copy-paste them. If you miss a digit from selecting wrong, you should get a more-or-less immediate exception. Why is this happening to people so often?
Reacted by Hans Brende, Sebastian Hädrich, Artem, Gabriele Tomberli and ZMRyan Cavanaugh (@RyanCavanaugh) to answer your question for my own use-cases, the ability to correctly type a UUID would be helpful mainly to ensure that UUIDs round-trip to the server without accidentally putting some other string identifier (whether that be some human-readable identifier, "code", or stringified serial ID) in the "id" field (which aligns with the whole point of using a typed language in the first place: fail at compile time instead of runtime).
Of course, that problem is easily solved by using opaque symbol tags as well, though it feels kind of hacky, especially when certain "special" uuids are hardcoded and must be cast to fit the opaque type.
A more compelling use-case in my mind is hex codes of a certain length, such as RGB or RGBA codes, especially when you are doing math on them and want to avoid writing extra code for error handling of non-valid inputs (or even worse: trying to support all possible formats to avoid error-handling).
Reacted by Sebastian Hädrich, Jordan Harband, Petter, Gabriele Tomberli, AverageHelper, ZM, Ed Galligan and Ryan PimiskernWhile this may seem very niche, I'd hope that composable Regex-validated string types could be supported (similar to how the
RegExpconstructor can be used to dynamically create a pattern and flags from strings) by providing an intrinsic type like the following:type Res = Regex<'[a-z]\\d', 'i'>; // ^? type Res = /[a-z]\d/i
would be cool if the inverse also worked:
type RegexObj<T> = T extends Regex<infer Pattern, infer Flags> ? { pattern: Pattern, flags: Flags } : never ; type Res = RegexObj</[a-z]\d/i>; // ^? type Res = { pattern: "[a-z]\\d"; flags: "i"; }
Reacted by Alejandro MeryFor the fixed-length subset of this (e.g. #52243, 20-byte vs 32-byte hex strings), there's a workaround that type-checks today (TS 5.4+, including 7.0), with no brands or generic helper functions at the use site:
type End = "" & String; // placeholder that only matches "" type C = string & {}; // placeholder that matches one character type Exactly<N extends number, S extends string = "", I extends 0[] = []> = I["length"] extends N ? `${S}${End}` : Exactly<N, `${S}${C}`, [...I, 0]>; type Address = `${bigint}` & `0x${Exactly<40>}`; type Hash = `${bigint}` & `0x${Exactly<64>}`; const addr: Address = "0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045"; // ok const typo: Address = "0xd8dA6BF26964aF9D7eEd9e03E53415D37aA9604"; // error declare const hash: Hash; const oops: Address = hash; // error
How it works:
- Two adjacent placeholders: the first one matches exactly one character.
"" & {}gets normalized into literal text, but"" & Stringstays a placeholder that only accepts"", so it anchors the end of the string. Without it, the last placeholder would match the rest of the input.${bigint}accepts0x…hex, so the intersection also restricts both types to hex.
Limits: per-character classes are limited to what a placeholder can express (e.g. any character, a digit via
${bigint}, no uppercase viaLowercase<string>), plus whole-string patterns via intersection, lengths top out around 999 because of the recursion limit, and "character" means a UTF-16 code unit in 5.x but a code point in 7.x.This isn't a substitute for regex types, but it does cover the "exact length + prefix" cases people keep asking for.
Reacted by Petter
This is a pickup of #6579. With the addition of #40336, a large number of those use cases have been addressed, but possibly some still remain.
Update 2023-04-11: Reviewed use cases and posted a write-up of our current evaluation
Search Terms
regex string types
Suggestion
Open question: For people who had upvoted #6579, what use cases still need addressing?
Note: Please keep discussion on-topic; moderation will be a bit heavier to avoid off-topic tangents
Examples
(please help)
Checklist
My suggestion meets these guidelines: