crypto: scrypt with params over maxmem leaves an OpenSSL error queued; the next createPrivateKey fails #66668

Version

v24.13.0 (OpenSSL 3.5.4)

Platform

Windows 11 x64 (10.0.22631). Not yet checked on Linux/macOS.

Subsystem

crypto

What steps will reproduce the bug?

import crypto from "node:crypto";

const { privateKey } = crypto.generateKeyPairSync("rsa", {
  modulusLength: 2048,
  privateKeyEncoding: { type: "pkcs8", format: "pem" },
  publicKeyEncoding: { type: "spki", format: "pem" },
});

crypto.createPrivateKey(privateKey); // ok

try {
  // 128 * r * (N + 2) > maxmem → ERR_CRYPTO_INVALID_SCRYPT_PARAMS (expected)
  crypto.scryptSync("pw", crypto.randomBytes(16), 32, { N: 131072, r: 32, p: 1, maxmem: 256 * 1024 * 1024 });
} catch (e) {
  console.log(e.code); // ERR_CRYPTO_INVALID_SCRYPT_PARAMS
}

crypto.createPrivateKey(privateKey);
// throws: ERR_OSSL_EVP_MEMORY_LIMIT_EXCEEDED
//   error:030000AC:digital envelope routines::memory limit exceeded

crypto.createPrivateKey(privateKey); // ok again

The asynchronous crypto.scrypt(..., cb) behaves the same way: its parameter check throws synchronously and leaves the
same error queued.

How often does it reproduce? Is there a required condition?

Always, when the scrypt parameters exceed maxmem (the memory check). A non-power-of-two N (also
ERR_CRYPTO_INVALID_SCRYPT_PARAMS) does not leave an error behind.

What is the expected behavior? Why is that the expected behavior?

The failed scrypt parameter check should not affect unrelated later crypto calls: createPrivateKey of a valid key
should succeed. The OpenSSL error queue should be cleared when the parameter check fails, as Node does elsewhere.

What do you see instead?

The first createPrivateKey (or other key-parsing call) after the failed scrypt throws the scrypt's
EVP_R_MEMORY_LIMIT_EXCEEDED. Through jsonwebtoken this surfaced as «secretOrPrivateKey must be an asymmetric key when
using RS256» (jsonwebtoken retries a key that fails createPrivateKey as an HMAC secret), which made it look like a
random signing failure under load.

Additional information

The likely source is the fix for #53291 (PR #53300): when ncrypto::checkScryptParams fails, src/crypto/crypto_scrypt.cc
(line ~92 on main) reads the OpenSSL error with ERR_peek_last_error() to build the message, but never pops or
clears it, so it stays queued for the next OpenSSL consumer on the thread. A ClearErrorOnReturn (or
ERR_get_error() / ERR_clear_error() after formatting) on that path should fix it. Same class as #45485
(X509Certificate.checkPrivateKey() with a non-matching key makes the next createPrivateKey() fail), #21281 and
#42793.