npm package appears corrupt

[saimonmoore]

Unable to install this package:

 yarn add --registry https://registry.npmjs.org/ kafkajs-lz4
yarn add v1.22.4
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
error https://registry.npmjs.org/lz4/-/lz4-0.6.3.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, link '/Users/saimon.moore/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/build/Release/lz4.node' -> '/Users/saimon.moore/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/lz4.node'"
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

[paambaati]

This seems to be an upstream issue — I can replicate this issue when I try to install the lz4 package.

$ yarn add lz4

yarn add v1.22.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error https://registry.yarnpkg.com/lz4/-/lz4-0.6.3.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, link '/Users/gp/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/build/Release/lz4.node' -> '/Users/gp/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/lz4.node'"

@saimonmoore Please open an issue in the lz4 project's issue tracker.

[saimonmoore]

Was already opened: pierrec/node-lz4#97

[leoparis89]

It's due to a breaking change in yarn v1.22.
Running yarn policies set-version 1.21 fixed the problem for me.

[paambaati]

@saimonmoore Can you confirm the the workaround posted above by @leoparis89 fixes this issue?

yarn policies set-version 1.21

[paambaati]

Can you folks try the latest v2 beta release? See #7 (comment)

yarn add [email protected]

[link2malkeet]

Hey paambati
We are getting a typeerror, we tried both version 1.2.1 and the latest beta - 2.0.0-beta:
TypeError: kafkajs_lz4_1.default is not a constructor
Thanks

[paambaati]

@link2malkeet Can you share the version of Node and the relevant piece of code where you’re using this package?

[link2malkeet]

Thanks for coming so quickly,
That's resolved, actually we followed the documentation and found the issue with the way we need to call codec on the instance of LZ4Codec
with this code snippet, we manage to create an instance of LZ4Codec
const LZ4Codec = require('kafkajs-lz4'); // import LZ4Codec fromkafkajs-lz4did not work. const LZ4Instance = new LZ4Codec(); CompressionCodecs[CompressionTypes.LZ4] = LZ4Instance.codec;
But
After this configuration, still we are getting the same old error saying
LZ4 compression not implemented.

Usage looks very simple but could not get it working yet. 😓
Thanks
Mel

[paambaati]

@link2malkeet This should work though, I'm not sure of the root-cause of your issue.

Can you follow the examples (or even our tests); they're fairly straightforward and are known to work.

If that doesn't work, I'm afraid we'll need to know more about your setup and your code.

[link2malkeet]

Well our setup is pretty straight forward. Before we create Producer and Consumer instances, we simply add the compression configuration as suggested so build and deployment works fine but when consumer listens to the kafka topic(which has messages compressed with LZ4), the subscription throws this error saying LZ4 compression not implemented
Node: v14.15.0
Yarn: v1.21.1

can you point out where we are supposed to add LZ4 configuration?

[paambaati]

where we are supposed to add LZ4 configuration?

@link2malkeet They're supposed to be added to the producer.send method. You can check our tests (see

kafkajs-lz4/test/index.test.ts

Lines 59 to 112 in 5efecce

	test('👩🏻‍🔬 Should compress and decompress real Kafka messages.', async (t) => {
	t.plan(2);
	const fixture = {
	topicName: 'topic-test',
	message: {
	key: Buffer.from('lorem'),
	value: Buffer.from(
	'Lorem ipsum dolor sit amet, consectetur adipisicing elit,' +
	'sed do eiusmod tempor incididunt ut labore et dolore magna ' +
	'aliqua. Ut enim ad minim veniam, quis nostrud exercitation' +
	'ullamco laboris nisi ut aliquip ex ea commodo consequat. ' +
	'Duis aute irure dolor in reprehenderit in voluptate velit ' +
	'esse cillum dolore eu fugiat nulla pariatur. Excepteur ' +
	'sint occaecat cupidatat non proident, sunt in culpa qui ' +
	'officia deserunt mollit anim id est laborum'
	),
	},
	};
	const kafka = new Kafka({
	brokers: [`${findKafkaLocation()}:9092`],
	clientId: 'kafkajs-lz4',
	logLevel: logLevel.NOTHING,
	});
	const producer = kafka.producer();
	const consumer = kafka.consumer({ groupId: 'lz4-group' });
	const lz4Codec = new LZ4Codec().codec;
	CompressionCodecs[CompressionTypes.LZ4] = lz4Codec;
	await producer.connect();
	await producer.send({
	topic: fixture.topicName,
	compression: CompressionTypes.LZ4,
	messages: [fixture.message],
	});
	await producer.disconnect();
	const messages: KafkaMessage[] = [];
	await consumer.connect();
	await consumer.subscribe({
	topic: fixture.topicName,
	fromBeginning: true,
	});
	// @ts-ignore
	consumer.run({ eachMessage: ({ message }) => messages.push(message as KafkaMessage) });
	await waitFor(() => messages.length >= 1, {
	maxWait: 60000,
	});
	const message = messages.pop() || { key: Buffer.from(''), value: Buffer.from('') };
	t.true(message.key.equals(fixture.message.key), 'Key should match.');
	t.true(message.value.equals(fixture.message.value), 'Value should match.');
	await consumer.disconnect();
	t.end();
	});

) and the official documentation (see https://kafka.js.org/docs/producing).

[link2malkeet]

Finally it worked.
Instead of using this library, we simple took the index.ts file source.

Seems this is culprit
there are 2 things exporting LZ4Codec
export default class LZ4Codec vs module.exports = LZ4Codec;

[link2malkeet]

Can you guys please update it with the change so that we could use this libraray with the required fix?

[paambaati]

@link2malkeet I'm sorry, but I don't think that's the right approach. I'd recommend you read up about CommonJS and ES6 imports and how they're different.

For a proper solution to your issue, like I've said before, I'd need more information — that includes your Node version, TS version (if you're using any), Babel config (if you're using any) and more relevant code (how you're importing this package, how/where you're using the produce & consume methods, etc.)

If you'd like to pursue this, I'd recommend you open a new issue as yours seems to be different from this.

[yayapao]

Same issues when i use nestjs monorepo.
I fininally solve it by use const LZ4Codec = require('kafkajs-lz4'), and here is my tsconfig.json:

{
  "compilerOptions": {
    "module": "CommonJS",
    "target": "ES2017"
    "Other configs": "..."
  }
}