You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
numcodecs.LZ4 not compatible with lz4.fram and cramjam.lz4 #816
While comparing LZ4 implementations, it was found that lz4.frame and cramjam.lz4 are compatible with each other, but numcodecs.LZ4 is not compatible with them.
From the tests :
Data compressed with lz4.frame can be decompressed by cramjam.lz4 and vice versa.
Data compressed with lz4.frame or cramjam.lz4 cannot be decompressed using numcodecs.LZ4.
Data compressed with numcodecs.LZ4 cannot be decompressed using lz4.frame or cramjam.lz4.
importcramjamimportnumcodecsimportosimportlz4.frameinitial_data=os.urandom(128)
compressed= {}
compressed["lz4"] =lz4.frame.compress(initial_data)
compressed["cramjam"] =bytes(cramjam.lz4.compress(initial_data))
compressed["numcodecs"] =numcodecs.LZ4().encode(initial_data)
libraries= ["lz4", "cramjam", "numcodecs"]
forcompressorinlibraries:
fordecompressorinlibraries:
print(f"Decompressing data compressed by {compressor} using {decompressor}... ", end="")
try:
ifdecompressor=="lz4":
decompressed=lz4.frame.decompress(compressed[compressor])
elifdecompressor=="cramjam":
decompressed=bytes(cramjam.lz4.decompress(compressed[compressor]))
elifdecompressor=="numcodecs":
decompressed=numcodecs.LZ4().decode(compressed[compressor])
assertinitial_data==decompressed, f"Decompression failed for {decompressor} with data compressed by {compressor}"print("Success!")
exceptExceptionase:
print(f"Failed: {e}")
Outputs :
Decompressing data compressed by lz4 using cramjam... Success!
Decompressing data compressed by lz4 using numcodecs... Failed: LZ4 decompression error: -13
Decompressing data compressed by cramjam using lz4... Success!
Decompressing data compressed by cramjam using cramjam... Success!
Decompressing data compressed by cramjam using numcodecs... Failed: LZ4 decompression error: -9
Decompressing data compressed by numcodecs using lz4... Failed: LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown
Decompressing data compressed by numcodecs using cramjam... Failed: LZ4 error: ERROR_frameType_unknown
Decompressing data compressed by numcodecs using numcodecs... Success!
The compressed output includes a 4-byte header storing the original size of the decompressed
data as a little-endian 32-bit integer.
whereas other libraries have an LZ4Frame which has more metadata. To use it, one needs lz4frame.h and lz4frame.c, but blosc doesn't include those (see here).
If anyone wants to work on this, here's a vibe-coded fix that you can use as inspiration ariostas@3195c4c
Activity
ariostas commentedon Mar 12, 2026
So it turns out that the issue is that the headers are different.
numcodecsjust has a 4-byte header for the uncompressed sizenumcodecs/numcodecs/lz4.pyx
Lines 61 to 64 in 408a367
whereas other libraries have an LZ4Frame which has more metadata. To use it, one needs lz4frame.h and lz4frame.c, but
bloscdoesn't include those (see here).If anyone wants to work on this, here's a vibe-coded fix that you can use as inspiration ariostas@3195c4c
Rachit931 commentedon Mar 18, 2026
I would like to take this issue
Will update as soon as I make progress.