From f6ba725e0a9db45dd5e363092d99b316c9540564 Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Tue, 9 May 2023 15:26:18 +0300 Subject: [PATCH 1/9] Error handling section --- README.md | 36 ++++---- sections/errorhandling/testingerrorflows.md | 91 ++++++++++----------- 2 files changed, 62 insertions(+), 65 deletions(-) diff --git a/README.md b/README.md index dc3f77459..3ad8dbb9e 100644 --- a/README.md +++ b/README.md @@ -70,12 +70,12 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [2.1 Use Async-Await or promises for async error handling](#-21-use-async-await-or-promises-for-async-error-handling)
-  [2.2 Use only the built-in Error object `#strategic`](#-22-use-only-the-built-in-error-object)
-  [2.3 Distinguish operational vs programmer errors `#strategic`](#-23-distinguish-operational-vs-programmer-errors)
+  [2.2 Extend the built-in Error object `#strategic #updated`](#-22-extend-the-built-in-error-object)
+  [2.3 Distinguish operational vs programmer errors `#strategic` `#updated`](#-23-distinguish-catastrophic-errors-from-operational-errors)
  [2.4 Handle errors centrally, not within a middleware `#strategic`](#-24-handle-errors-centrally-not-within-a-middleware)
-  [2.5 Document API errors using Swagger or GraphQL `#modified-recently`](#-25-document-api-errors-using-swagger-or-graphql)
+  [2.5 Document API errors using OpenAPI or GraphQL `#updated`](#-25-document-api-errors-using-openapi-or -graphql)
  [2.6 Exit the process gracefully when a stranger comes to town `#strategic`](#-26-exit-the-process-gracefully-when-a-stranger-comes-to-town)
-  [2.7 Use a mature logger to increase error visibility](#-27-use-a-mature-logger-to-increase-error-visibility)
+  [2.7 Use a mature logger to increase errors visibility `#updated`](#-27-use-a-mature-logger-to-increase-errors-visibility)
  [2.8 Test error flows using your favorite test framework](#-28-test-error-flows-using-your-favorite-test-framework)
  [2.9 Discover errors and downtime using APM products](#-29-discover-errors-and-downtime-using-apm-products)
  [2.10 Catch unhandled promise rejections `#modified-recently`](#-210-catch-unhandled-promise-rejections)
@@ -309,7 +309,7 @@ my-system ## ![✔] 2.1 Use Async-Await or promises for async error handling -**TL;DR:** Handling async errors in callback style is probably the fastest way to hell (a.k.a the pyramid of doom). The best gift you can give to your code is using a reputable promise library or async-await instead which enables a much more compact and familiar code syntax like try-catch +**TL;DR:** Handling async errors in callback style is probably the fastest way to hell (a.k.a the pyramid of doom). The best gift you can give to your code is using Promises with async-await which enables a much more compact and familiar code syntax like try-catch **Otherwise:** Node.js callback style, function(err, response), is a promising way to un-maintainable code due to the mix of error handling with casual code, excessive nesting, and awkward coding patterns @@ -317,9 +317,9 @@ my-system

-## ![✔] 2.2 Use only the built-in Error object +## ![✔] 2.2 Extend the built-in Error object -**TL;DR:** Many throw errors as a string or as some custom type – this complicates the error handling logic and the interoperability between modules. Whether you reject a promise, throw an exception or emit an error – using only the built-in Error object (or an object that extends the built-in Error object) will increase uniformity and prevent loss of information. There is `no-throw-literal` ESLint rule that strictly checks that (although it has some [limitations](https://eslint.org/docs/rules/no-throw-literal) which can be solved when using TypeScript and setting the `@typescript-eslint/no-throw-literal` rule) +**TL;DR:** Some libraries throw errors as a string or as some custom type – this complicates the error handling logic and the interoperability between modules. Instead, create app error object/class that extends the built-in Error object and use it whenever rejecting, throwing or emitting an error. The app error should add useful imperative properties like the error name/code and isCatastrophic. By doing so, all errors have a unified structure and support better error handling .There is `no-throw-literal` ESLint rule that strictly checks that (although it has some [limitations](https://eslint.org/docs/rules/no-throw-literal) which can be solved when using TypeScript and setting the `@typescript-eslint/no-throw-literal` rule) **Otherwise:** When invoking some component, being uncertain which type of errors come in return – it makes proper error handling much harder. Even worse, using custom types to describe errors might lead to loss of critical error information like the stack trace! @@ -327,11 +327,11 @@ my-system

-## ![✔] 2.3 Distinguish operational vs programmer errors +## ![✔] 2.3 Distinguish catastrophic errors from operational errors -**TL;DR:** Operational errors (e.g. API received an invalid input) refer to known cases where the error impact is fully understood and can be handled thoughtfully. On the other hand, programmer error (e.g. trying to read an undefined variable) refers to unknown code failures that dictate to gracefully restart the application +**TL;DR:** Operational errors (e.g. API received an invalid input) refer to known cases where the error impact is fully understood and can be handled thoughtfully. On the other hand, catastrophic error (also known as programmer errors) refers to unusual code failures that dictate to gracefully restart the application -**Otherwise:** You may always restart the application when an error appears, but why let ~5000 online users down because of a minor, predicted, operational error? The opposite is also not ideal – keeping the application up when an unknown issue (programmer error) occurred might lead to an unpredicted behavior. Differentiating the two allows acting tactfully and applying a balanced approach based on the given context +**Otherwise:** You may always restart the application when an error appears, but why let ~5000 online users down because of a minor, predicted, operational error? The opposite is also not ideal – keeping the application up when an unknown catastrophic issue (programmer error) occurred might lead to an unpredicted behavior. Differentiating the two allows acting tactfully and applying a balanced approach based on the given context 🔗 [**Read More: operational vs programmer error**](./sections/errorhandling/operationalvsprogrammererror.md) @@ -339,7 +339,7 @@ my-system ## ![✔] 2.4 Handle errors centrally, not within a middleware -**TL;DR:** Error handling logic such as mail to admin and logging should be encapsulated in a dedicated and centralized object that all endpoints (e.g. Express middleware, cron jobs, unit-testing) call when an error comes in +**TL;DR:** Error handling logic such as logging, deciding whether to crash and monitoring metrics should be encapsulated in a dedicated and centralized object that all entry-points (e.g. APIs, cron jobs, scheduled jobs) call when an error comes in **Otherwise:** Not handling errors within a single place will lead to code duplication and probably to improperly handled errors @@ -347,9 +347,9 @@ my-system

-## ![✔] 2.5 Document API errors using Swagger or GraphQL +## ![✔] 2.5 Document API errors using OpenAPI or GraphQL -**TL;DR:** Let your API callers know which errors might come in return so they can handle these thoughtfully without crashing. For RESTful APIs, this is usually done with documentation frameworks like Swagger. If you're using GraphQL, you can utilize your schema and comments as well. +**TL;DR:** Let your API callers know which errors might come in return so they can handle these thoughtfully without crashing. For RESTful APIs, this is usually done with documentation frameworks like OpenAPI. If you're using GraphQL, you can utilize your schema and comments as well **Otherwise:** An API client might decide to crash and restart only because it received back an error it couldn’t understand. Note: the caller of your API might be you (very typical in a microservice environment) @@ -359,7 +359,7 @@ my-system ## ![✔] 2.6 Exit the process gracefully when a stranger comes to town -**TL;DR:** When an unknown error occurs (a developer error, see best practice 2.3) - there is uncertainty about the application healthiness. Common practice suggests restarting the process carefully using a process management tool like [Forever](https://www.npmjs.com/package/forever) or [PM2](http://pm2.keymetrics.io/) +**TL;DR:** When an unknown error occurs (catastrophic error, see best practice 2.3) - there is uncertainty about the application healthiness. In this case, there is no escape from making the error observable, shutting off connections and exiting the process. Any reputable runtime framework like Dockerized services or cloud serverless solutions will take care to restart **Otherwise:** When an unfamiliar exception occurs, some object might be in a faulty state (e.g. an event emitter which is used globally and not firing events anymore due to some internal failure) and all future requests might fail or behave crazily @@ -367,9 +367,9 @@ my-system

-## ![✔] 2.7 Use a mature logger to increase error visibility +## ![✔] 2.7 Use a mature logger to increase errors visibility -**TL;DR:** A set of mature logging tools like [Pino](https://github.com/pinojs/pino) or [Log4js](https://www.npmjs.com/package/log4js), will speed-up error discovery and understanding. So forget about console.log +**TL;DR:** A robust logging tools like [Pino](https://github.com/pinojs/pino) or [Winston](https://github.com/winstonjs/winston) increases the errors visibility using features like log-levels, pretty print coloring and more. Console.log lacks these imperative features and should be avoided. The best in class logger allows attaching custom useful properties to log entries with minimized serialization performance penalty. Developers should write logs to `stdout` and let the infrastructure pipe the stream to the appropriate log aggregator **Otherwise:** Skimming through console.logs or manually through messy text file without querying tools or a decent log viewer might keep you busy at work until late @@ -379,7 +379,7 @@ my-system ## ![✔] 2.8 Test error flows using your favorite test framework -**TL;DR:** Whether professional automated QA or plain manual developer testing – Ensure that your code not only satisfies positive scenarios but also handles and returns the right errors. Testing frameworks like Mocha & Chai can handle this easily (see code examples within the "Gist popup") +**TL;DR:** Whether professional automated QA or plain manual developer testing – Ensure that your code not only satisfies positive scenarios but also handles and returns the right errors. On top of this, simulate deeper error flows like uncaught exceptions an ensure that the error handler treat these properly (see code examples within the "read more" section) **Otherwise:** Without testing, whether automatically or manually, you can’t rely on your code to return the right errors. Without meaningful errors – there’s no error handling @@ -409,7 +409,7 @@ my-system ## ![✔] 2.11 Fail fast, validate arguments using a dedicated library -**TL;DR:** Assert API input to avoid nasty bugs that are much harder to track later. The validation code is usually tedious unless you are using a very cool helper library like [ajv](https://www.npmjs.com/package/ajv) and [Joi](https://www.npmjs.com/package/joi) +**TL;DR:** Assert API input to avoid nasty bugs that are much harder to track later. The validation code is usually tedious unless you are using a modern validation library like [ajv](https://www.npmjs.com/package/ajv), [zod](https://github.com/colinhacks/zod), or [typebox](https://github.com/sinclairzx81/typebox) **Otherwise:** Consider this – your function expects a numeric argument “Discount” which the caller forgets to pass, later on, your code checks if Discount!=0 (amount of allowed discount is greater than zero), then it will allow the user to enjoy a discount. OMG, what a nasty bug. Can you see it? diff --git a/sections/errorhandling/testingerrorflows.md b/sections/errorhandling/testingerrorflows.md index 552ca1f53..ac03f4146 100644 --- a/sections/errorhandling/testingerrorflows.md +++ b/sections/errorhandling/testingerrorflows.md @@ -10,72 +10,69 @@ Testing ‘happy’ paths is no better than testing failures. Good testing code Javascript ```javascript -describe('Facebook chat', () => { - it('Notifies on new chat message', () => { +describe("Facebook chat", () => { + it("Notifies on new chat message", () => { const chatService = new chatService(); chatService.participants = getDisconnectedParticipants(); - expect(chatService.sendMessage.bind({ message: 'Hi' })).to.throw(ConnectionError); + expect(chatService.sendMessage.bind({ message: "Hi" })).to.throw(ConnectionError); }); }); ``` + +### Code example: ensuring API returns the right HTTP error code and log properly +
-Typescript +Javascript -```typescript -describe('Facebook chat', () => { - it('Notifies on new chat message', () => { - const chatService = new chatService(); - chatService.participants = getDisconnectedParticipants(); - expect(chatService.sendMessage.bind({ message: 'Hi' })).to.throw(ConnectionError); +```javascript +test("When exception is throw during request, Then logger reports the mandatory fields", async () => { + //Arrange + const orderToAdd = { + userId: 1, + productId: 2, + }; + + sinon + .stub(OrderRepository.prototype, "addOrder") + .rejects(new AppError("saving-failed", "Order could not be saved", 500)); + const loggerDouble = sinon.stub(logger, "error"); + + //Act + const receivedResponse = await axiosAPIClient.post("/order", orderToAdd); + + //Assert + expect(receivedResponse.status).toBe(500); + expect(loggerDouble.lastCall.firstArg).toMatchObject({ + name: "saving-failed", + status: 500, + stack: expect.any(String), + message: expect.any(String), }); }); ``` +
-### Code example: ensuring API returns the right HTTP error code +### Code example: ensuring that are uncaught exceptions are handled as well
Javascript ```javascript -it('Creates new Facebook group', () => { - const invalidGroupInfo = {}; - return httpRequest({ - method: 'POST', - uri: 'facebook.com/api/groups', - resolveWithFullResponse: true, - body: invalidGroupInfo, - json: true - }).then((response) => { - expect.fail('if we were to execute the code in this block, no error was thrown in the operation above') - }).catch((response) => { - expect(400).to.equal(response.statusCode); - }); -}); -``` -
+test("When unhandled exception is throw, Then the logger reports correctly", async () => { + //Arrange + await api.startWebServer(); + const loggerDouble = sinon.stub(logger, "error"); + const errorToThrow = new Error("An error that wont be caught 😳"); -
-Typescript - -```typescript -it('Creates new Facebook group', async () => { - let invalidGroupInfo = {}; - try { - const response = await httpRequest({ - method: 'POST', - uri: 'facebook.com/api/groups', - resolveWithFullResponse: true, - body: invalidGroupInfo, - json: true - }) - // if we were to execute the code in this block, no error was thrown in the operation above - expect.fail('The request should have failed') - } catch(response) { - expect(400).to.equal(response.statusCode); - } + //Act + process.emit("uncaughtException", errorToThrow); + + // Assert + expect(loggerDouble.calledWith(errorToThrow)); }); ``` -
\ No newline at end of file + + From f8b1810fa75b45a31f43a7c62212584dea1b4fd3 Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Tue, 9 May 2023 21:40:51 +0300 Subject: [PATCH 2/9] Code quality section --- README.md | 41 +++++++++++++++++++++++------------------ 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 3ad8dbb9e..e2f130b0b 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [3.10 Use the === operator](#-310-use-the--operator)
  [3.11 Use Async Await, avoid callbacks `#strategic`](#-311-use-async-await-avoid-callbacks)
  [3.12 Use arrow function expressions (=>)](#-312-use-arrow-function-expressions-)
-  [3.13 Avoid effects outside of functions #new](#-313-avoid-effects-outside-of-functions)
+  [3.13 Avoid effects outside of functions `#new`](#-313-avoid-effects-outside-of-functions)
@@ -113,15 +113,15 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [4.1 At the very least, write API (component) testing `#strategic`](#-41-at-the-very-least-write-api-component-testing)
  [4.2 Include 3 parts in each test name `#new`](#-42-include-3-parts-in-each-test-name)
  [4.3 Structure tests by the AAA pattern `#strategic`](#-43-structure-tests-by-the-aaa-pattern)
-  [4.4 Ensure Node version is unified #new](#-44-ensure-node-version-is-unified)
+  [4.4 Ensure Node version is unified `#new`](#-44-ensure-node-version-is-unified)
  [4.5 Avoid global test fixtures and seeds, add data per-test `#strategic`](#-45-avoid-global-test-fixtures-and-seeds-add-data-per-test)
  [4.6 Tag your tests `#advanced`](#-46-tag-your-tests)
  [4.7 Check your test coverage, it helps to identify wrong test patterns](#-47-check-your-test-coverage-it-helps-to-identify-wrong-test-patterns)
  [4.8 Use production-like environment for e2e testing](#-48-use-production-like-environment-for-e2e-testing)
  [4.9 Refactor regularly using static analysis tools](#-49-refactor-regularly-using-static-analysis-tools)
-  [4.10 Mock responses of external HTTP services #advanced #new](#-410-mock-responses-of-external-http-services)
+  [4.10 Mock responses of external HTTP services #advanced `#new` `#advanced`](#-410-mock-responses-of-external-http-services)
  [4.11 Test your middlewares in isolation](#-411-test-your-middlewares-in-isolation)
-  [4.12 Specify a port in production, randomize in testing #new](#-412-specify-a-port-in-production-randomize-in-testing)
+  [4.12 Specify a port in production, randomize in testing `#new`](#-412-specify-a-port-in-production-randomize-in-testing)
  [4.13 Test the five possible outcomes #strategic #new](#-413-test-the-five-possible-outcomes)
@@ -437,7 +437,7 @@ especially if the cause of the abnormal behavior is inside of the missing functi ## ![✔] 3.1 Use ESLint -**TL;DR:** [ESLint](https://eslint.org) is the de-facto standard for checking possible code errors and fixing code style, not only to identify nitty-gritty spacing issues but also to detect serious code anti-patterns like developers throwing errors without classification. Though ESLint can automatically fix code styles, other tools like [prettier](https://www.npmjs.com/package/prettier) and [beautify](https://www.npmjs.com/package/js-beautify) are more powerful in formatting the fix and work in conjunction with ESLint +**TL;DR:** [ESLint](https://eslint.org) is the de-facto standard for checking possible code errors and fixing code style, not only to identify nitty-gritty spacing issues but also to detect serious code anti-patterns like developers throwing errors without classification. Though ESLint can automatically fix code styles, other tools like [prettier](https://www.npmjs.com/package/prettier) are more powerful in formatting the fix and work in conjunction with ESLint **Otherwise:** Developers will focus on tedious spacing and line-width concerns and time might be wasted overthinking the project's code style @@ -445,9 +445,9 @@ especially if the cause of the abnormal behavior is inside of the missing functi

-## ![✔] 3.2 Node.js specific plugins +## ![✔] 3.2 Use Node.js eslint extension plugins -**TL;DR:** On top of ESLint standard rules that cover vanilla JavaScript, add Node.js specific plugins like [eslint-plugin-node](https://www.npmjs.com/package/eslint-plugin-node), [eslint-plugin-mocha](https://www.npmjs.com/package/eslint-plugin-mocha) and [eslint-plugin-node-security](https://www.npmjs.com/package/eslint-plugin-security) +**TL;DR:** On top of ESLint standard rules that cover vanilla JavaScript, add Node.js specific plugins like [eslint-plugin-node](https://www.npmjs.com/package/eslint-plugin-node), [eslint-plugin-mocha](https://www.npmjs.com/package/eslint-plugin-mocha) and [eslint-plugin-node-security](https://www.npmjs.com/package/eslint-plugin-security), [eslint-plugin-require](https://www.npmjs.com/package/eslint-plugin-require), [/eslint-plugin-jest](https://www.npmjs.com/package/eslint-plugin-jest) and other useful rules **Otherwise:** Many faulty Node.js code patterns might escape under the radar. For example, developers might require(variableAsPath) files with a variable given as a path which allows attackers to execute any JS script. Node.js linters can detect such patterns and complain early @@ -588,22 +588,27 @@ function doSomething() {

-## ![✔] 3.9 Require modules by folders, as opposed to the files directly +## ![✔] 3.9 Set an explicit entry point per module/folder -**TL;DR:** When developing a module/library in a folder, place an index.js file that exposes the module's internals so every consumer will pass through it. This serves as an 'interface' to your module and eases future changes without breaking the contract +**TL;DR:** When developing a module/library, set an explicit root file that exports the public and interesting code. Discourage the client code from importing deep files and becoming familiar with the internal structure. With commonjs (require), this can be done with an index.js file at the folder's root or the package.json.main field. With ESM (import), if a package.json exists on the root, the field "exports" allow specifying the module's root file. If no package.json exists, you may put an index.js file on the root which re-exports all the public functionality -**Otherwise:** Changing the internal structure of files or the signature may break the interface with clients +**Otherwise:** Having an explicit root file acts like a public 'interface' that encapsulates the internal, directs the caller to the public code and facilitates future changes without breaking the contract -### 3.9 Code example +### 3.9 Code example - avoid coupling the client to the module structure ```javascript -// Do -module.exports.SMSProvider = require("./SMSProvider"); -module.exports.SMSNumberResolver = require("./SMSNumberResolver"); +// Avoid: client has deep familiarity with the internals -// Avoid -module.exports.SMSProvider = require("./SMSProvider/SMSProvider.js"); -module.exports.SMSNumberResolver = require("./SMSNumberResolver/SMSNumberResolver.js"); +// Client code +const SMSWithMedia = require("./SMSProvider/providers/media/media-provider.js"); + +// Better: explicitly export the public functions + +//index.js, module code +module.exports.SMSWithMedia = require("./SMSProvider/providers/media/media-provider.js"); + +// Client code +const { SMSWithMedia } = require("./SMSProvider"); ```

@@ -667,7 +672,7 @@ All statements above will return false if used with `===` # `4. Testing And Overall Quality Practices` -\_We have dedicated guides for testing, see below. The practices here are a brief summary of these guides +\_We have dedicated guides for testing, see below. The best practices list here is a brief summary of these guides a. [JavaScript testing best practices](https://github.com/goldbergyoni/javascript-testing-best-practices) b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodejs-integration-tests-best-practices) From 235516cc9f8f223b611a83bd0f6d31f3015fb063 Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Wed, 10 May 2023 11:26:38 +0300 Subject: [PATCH 3/9] Production practices --- README.md | 38 +++++++-------- sections/production/lockdependencies.md | 64 ++++++++----------------- sections/production/productioncode.md | 17 +++---- 3 files changed, 48 insertions(+), 71 deletions(-) diff --git a/README.md b/README.md index e2f130b0b..24d97b520 100644 --- a/README.md +++ b/README.md @@ -806,7 +806,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej

-## ![✔] 5.2. Increase transparency using smart logging +## ![✔] 5.2. Increase the observability using smart logging **TL;DR:** Logs can be a dumb warehouse of debug statements or the enabler of a beautiful dashboard that tells the story of your app. Plan your logging platform from day 1: how logs are collected, stored and analyzed to ensure that the desired information (e.g. error rate, following an entire transaction through services and servers, etc) can really be extracted @@ -818,7 +818,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.3. Delegate anything possible (e.g. gzip, SSL) to a reverse proxy -**TL;DR:** Node is awfully bad at doing CPU intensive tasks like gzipping, SSL termination, etc. You should use ‘real’ middleware services like nginx, HAproxy or cloud vendor services instead +**TL;DR:** Node is quite bad at doing CPU intensive tasks like gzipping, SSL termination, etc. You should use specialized infrastructure like nginx, HAproxy or cloud vendor services instead **Otherwise:** Your poor single thread will stay busy doing infrastructural tasks instead of dealing with your application core and performance will degrade accordingly @@ -828,7 +828,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.4. Lock dependencies -**TL;DR:** Your code must be identical across all environments, but amazingly npm lets dependencies drift across environments by default – when you install packages at various environments it tries to fetch packages’ latest patch version. Overcome this by using npm config files, .npmrc, that tell each environment to save the exact (not the latest) version of each package. Alternatively, for finer grained control use `npm shrinkwrap`. \*Update: as of NPM5, dependencies are locked by default. The new package manager in town, Yarn, also got us covered by default +**TL;DR:** Your code must be identical across all environments, but without a special lockfile npm lets dependencies drift across environments. Ensure to commit your package-lock.json so all the environments will be identical **Otherwise:** QA will thoroughly test the code and approve a version that will behave differently in production. Even worse, different servers in the same production cluster might run different code @@ -838,7 +838,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.5. Guard process uptime using the right tool -**TL;DR:** The process must go on and get restarted upon failures. For simple scenarios, process management tools like PM2 might be enough but in today's ‘dockerized’ world, cluster management tools should be considered as well +**TL;DR:** The process must go on and get restarted upon failures. Modern runtime platforms like Docker-ized platforms (e.g. Kubernetes), and Serverless take care for this automatically. When the app is hosted on a bare metal server, one must take care for a process management tools like [systemd](https://systemd.io/). Avoid including a custom process management tool in a modern platform that monitor an app instance (e.g., Kubernetes) - doing so will hide failures from the infrastructure. When the underlying infrastructure is not aware of errors, it can't perform useful mitigation steps like re-placing the instance in a different location **Otherwise:** Running dozens of instances without a clear strategy and too many tools together (cluster management, docker, PM2) might lead to DevOps chaos @@ -848,7 +848,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.6. Utilize all CPU cores -**TL;DR:** At its basic form, a Node app runs on a single CPU core while all others are left idling. It’s your duty to replicate the Node process and utilize all CPUs – For small-medium apps you may use Node Cluster or PM2. For a larger app consider replicating the process using some Docker cluster (e.g. K8S, ECS) or deployment scripts that are based on Linux init system (e.g. systemd) +**TL;DR:** At its basic form, a Node app runs on a single CPU core while all others are left idling. It’s your duty to replicate the Node process and utilize all CPUs. Most of the modern run-times platform (e.g., Kubernetes) allow replicating instances of the app but they won't verify that all cores are utilized - this is your duty. If the app is hosted on a bare server, it's also your duty to use some process replication solution (e.g. systemd) **Otherwise:** Your app will likely utilize only 25% of its available resources(!) or even less. Note that a typical server has 4 CPU cores or more, naive deployment of Node.js utilizes only 1 (even using PaaS services like AWS beanstalk!) @@ -866,9 +866,9 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej

-## ![✔] 5.8. Discover errors and downtime using APM products +## ![✔] 5.8. Discover the unknowns using APM products -**TL;DR:** Application monitoring and performance products (a.k.a. APM) proactively gauge codebase and API so they can auto-magically go beyond traditional monitoring and measure the overall user-experience across services and tiers. For example, some APM products can highlight a transaction that loads too slow on the end-user's side while suggesting the root cause +**TL;DR:** Consider adding another safety layer to the production stack - APM. While the majority of symptoms and causes can be detected using traditional monitoring techniques, in a distributed system there is more than meets the eye. Application monitoring and performance products (a.k.a. APM) can auto-magically go beyond traditional monitoring and provide additional layer of discovery and developer-experience. For example, some APM products can highlight a transaction that loads too slow on the **end-user's side** while suggesting the root cause. APMs also provide more context for developers who try to troubleshoot a log error by showing what was the server busy with when the error occurred. To name a few example **Otherwise:** You might spend great effort on measuring API performance and downtimes, probably you’ll never be aware which is your slowest code parts under real-world scenario and how these affect the UX @@ -878,7 +878,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.9. Make your code production-ready -**TL;DR:** Code with the end in mind, plan for production from day 1. This sounds a bit vague so I’ve compiled a few development tips that are closely related to production maintenance (click Gist below) +**TL;DR:** Code with the end in mind, plan for production from day 1. This sounds a bit vague so I’ve compiled a few development tips that are closely related to production maintenance (click 'Read More') **Otherwise:** A world champion IT/DevOps guy won’t save a system that is badly written @@ -898,7 +898,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.11. Get your frontend assets out of Node -**TL;DR:** Serve frontend content using dedicated middleware (nginx, S3, CDN) because Node performance really gets hurt when dealing with many static files due to its single-threaded model +**TL;DR:** Serve frontend content using a specialized infrastructure (nginx, S3, CDN) because Node performance gets hurt when dealing with many static files due to its single-threaded model. One exception to this guideline is when doing server-side rendering **Otherwise:** Your single Node thread will be busy streaming hundreds of html/images/angular/react files instead of allocating all its resources for the task it was born for – serving dynamic content @@ -906,9 +906,9 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej

-## ![✔] 5.12. Be stateless, kill your servers almost every day +## ![✔] 5.12. Strive to be stateless -**TL;DR:** Store any type of data (e.g. user sessions, cache, uploaded files) within external data stores. Consider ‘killing’ your servers periodically or use ‘serverless’ platform (e.g. AWS Lambda) that explicitly enforces a stateless behavior +**TL;DR:** Store any type of _data_ (e.g. user sessions, cache, uploaded files) within external data stores. When the app holds data in-process this adds additional layer of maintenance complexity like routing users to the same instance and higher cost of restarting a process. To enforce and encourage a stateless approach, most modern runtime platforms allows 'reapp-ing' instances periodically **Otherwise:** Failure at a given server will result in application downtime instead of just killing a faulty machine. Moreover, scaling-out elasticity will get more challenging due to the reliance on a specific server @@ -918,7 +918,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.13. Use tools that automatically detect vulnerabilities -**TL;DR:** Even the most reputable dependencies such as Express have known vulnerabilities (from time to time) that can put a system at risk. This can be easily tamed using community and commercial tools that constantly check for vulnerabilities and warn (locally or at GitHub), some can even patch them immediately +**TL;DR:** Even the most reputable dependencies such as Express have known vulnerabilities (from time to time) that can put a system at risk. This can be easily be tamed using community and commercial tools that constantly check for vulnerabilities and warn (locally or at GitHub), some can even patch them immediately **Otherwise:** Keeping your code clean from vulnerabilities without dedicated tools will require you to constantly follow online publications about new threats. Quite tedious @@ -928,9 +928,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.14. Assign a transaction id to each log statement -Also known as correlation id / transit id / tracing id / request id / request context / etc. - -**TL;DR:** Assign the same identifier, transaction-id: {some value}, to each log entry within a single request. Then when inspecting errors in logs, easily conclude what happened before and after. Until version 14 of Node, this was not easy to achieve due to Node's async nature, but since AsyncLocalStorage came to town, this became possible and easy than ever. see code examples inside +**TL;DR:** Assign the same identifier, transaction-id: uuid(), to each log entry within a single request (also known as correlation-id/tracing-id/request-context). Then when inspecting errors in logs, easily conclude what happened before and after. Node has a built-in mechanism, [AsyncLocalStorage](https://nodejs.org/api/async_context.html), for keeping the same context across asynchronous calls. see code examples inside **Otherwise:** Looking at a production error log without the context – what happened before – makes it much harder and slower to reason about the issue @@ -940,9 +938,9 @@ Also known as correlation id / transit id / tracing id / request id / request co ## ![✔] 5.15. Set `NODE_ENV=production` -**TL;DR:** Set the environment variable `NODE_ENV` to ‘production’ or ‘development’ to flag whether production optimizations should get activated – many npm packages determine the current environment and optimize their code for production +**TL;DR:** Set the environment variable `NODE_ENV` to ‘production’ or ‘development’ to flag whether production optimizations should get activated – some npm packages determine the current environment and optimize their code for production -**Otherwise:** Omitting this simple property might greatly degrade performance. For example, when using Express for server-side rendering omitting `NODE_ENV` makes it slower by a factor of three! +**Otherwise:** Omitting this simple property might greatly degrade performance when dealing with some specific libraries like Express server-side rendering 🔗 [**Read More: Set NODE_ENV=production**](./sections/production/setnodeenv.md) @@ -966,11 +964,11 @@ Also known as correlation id / transit id / tracing id / request id / request co

-## ![✔] 5.18. Don't route logs within the app +## ![✔] 5.18. Log to stdout, avoid specifying log destination within the app **TL;DR:** Log destinations should not be hard-coded by developers within the application code, but instead should be defined by the execution environment the application runs in. Developers should write logs to `stdout` using a logger utility and then let the execution environment (container, server, etc.) pipe the `stdout` stream to the appropriate destination (i.e. Splunk, Graylog, ElasticSearch, etc.). -**Otherwise:** Application handling log routing === hard to scale, loss of logs, poor separation of concerns +**Otherwise:** If developers set the log routing, less flexibility is left for the ops professional who wishes to customize it. Beyond this, if the app tries to log directly to a remote location (e.g., Elastic Search), in case of panic or crash - further logs that might explain the problem won't arrive 🔗 [**Read More: Log Routing**](./sections/production/logrouting.md) @@ -978,7 +976,7 @@ Also known as correlation id / transit id / tracing id / request id / request co ## ![✔] 5.19. Install your packages with `npm ci` -**TL;DR:** You have to be sure that production code uses the exact version of the packages you have tested it with. Run `npm ci` to strictly do a clean install of your dependencies matching package.json and package-lock.json. Using this command is recommended in automated environments such as continuous integration pipelines. +**TL;DR:** Run `npm ci` to strictly do a clean install of your dependencies matching package.json and package-lock.json. Obviously production code must use the exact version of the packages that were used for testing. While package-lock.json file sets strict version for dependencies, in case of mismatch with the file package.json, the command 'npm install' will treat package.json as the source of truth. On the other hands, the command 'npm ci' will exit with error in case of mismatch between these files **Otherwise:** QA will thoroughly test the code and approve a version that will behave differently in production. Even worse, different servers in the same production cluster might run different code. diff --git a/sections/production/lockdependencies.md b/sections/production/lockdependencies.md index 1698c9bbb..61800062f 100644 --- a/sections/production/lockdependencies.md +++ b/sections/production/lockdependencies.md @@ -4,9 +4,7 @@ ### One Paragraph Explainer -Your code depends on many external packages, let’s say it ‘requires’ and use momentjs-2.1.4, then by default when you deploy to production npm might fetch momentjs 2.1.5 which unfortunately brings some new bugs to the table. Using npm config files and the argument ```–save-exact=true``` instructs npm to refer to the *exact* same version that was installed so the next time you run ```npm install``` (in production or within a Docker container you plan to ship forward for testing) the same dependent version will be fetched. An alternative and popular approach is using a `.shrinkwrap` file (easily generated using npm) that states exactly which packages and versions should be installed so no environment can get tempted to fetch newer versions than expected. - -* **Update:** as of npm 5, dependencies are locked automatically using .shrinkwrap. Yarn, an emerging package manager, also locks down dependencies by default. +Your code depends on many external packages, let’s say it ‘requires’ and use momentjs-2.1.4, then by default when you deploy to production npm might fetch momentjs 2.1.5 which unfortunately brings some new bugs to the table. Using npm config files and the argument `–save-exact=true` instructs npm to refer to the _exact_ same version that was installed so the next time you run `npm install` (in production or within a Docker container you plan to ship forward for testing) the same dependent version will be fetched. Due to this, starting from npm version 5 a package-lock.json file is generated in every install. This lock file pins all the dependencies and child dependencies versions. When the file is committed, any future install the gets a copy of the app will install the same dependencies version

@@ -19,51 +17,31 @@ save-exact:true

-### Code example: shrinkwrap.json file that distills the exact dependency tree - -```json -{ - "name": "A", - "dependencies": { - "B": { - "version": "0.0.1", - "dependencies": { - "C": { - "version": "0.1.0" - } - } - } - } -} -``` - -

- ### Code example: npm 5 dependencies lock file – package-lock.json ```json { - "name": "package-name", - "version": "1.0.0", - "lockfileVersion": 1, - "dependencies": { - "cacache": { - "version": "9.2.6", - "resolved": "https://registry.npmjs.org/cacache/-/cacache-9.2.6.tgz", - "integrity": "sha512-YK0Z5Np5t755edPL6gfdCeGxtU0rcW/DBhYhYVDckT+7AFkCCtedf2zru5NRbBLFk6e7Agi/RaqTOAfiaipUfg==" - }, - "duplexify": { - "version": "3.5.0", - "resolved": "https://registry.npmjs.org/duplexify/-/duplexify-3.5.0.tgz", - "integrity": "sha1-GqdzAC4VeEV+nZ1KULDMquvL1gQ=", - "dependencies": { - "end-of-stream": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.0.0.tgz", - "integrity": "sha1-1FlucCc0qT5A6a+GQxnqvZn/Lw4=" - } - } + "name": "package-name", + "version": "1.0.0", + "lockfileVersion": 1, + "dependencies": { + "cacache": { + "version": "9.2.6", + "resolved": "https://registry.npmjs.org/cacache/-/cacache-9.2.6.tgz", + "integrity": "sha512-YK0Z5Np5t755edPL6gfdCeGxtU0rcW/DBhYhYVDckT+7AFkCCtedf2zru5NRbBLFk6e7Agi/RaqTOAfiaipUfg==" + }, + "duplexify": { + "version": "3.5.0", + "resolved": "https://registry.npmjs.org/duplexify/-/duplexify-3.5.0.tgz", + "integrity": "sha1-GqdzAC4VeEV+nZ1KULDMquvL1gQ=", + "dependencies": { + "end-of-stream": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.0.0.tgz", + "integrity": "sha1-1FlucCc0qT5A6a+GQxnqvZn/Lw4=" } + } } + } } ``` diff --git a/sections/production/productioncode.md b/sections/production/productioncode.md index a86047312..6fb624a6c 100644 --- a/sections/production/productioncode.md +++ b/sections/production/productioncode.md @@ -6,11 +6,12 @@ Following is a list of development tips that greatly affect the production maintenance and stability: -* The twelve-factor guide – Get familiar with the [Twelve factors](https://12factor.net/) guide -* Be stateless – Save no data locally on a specific web server (see separate bullet – ‘Be Stateless’) -* Cache – Utilize cache heavily, yet never fail because of cache mismatch -* Test memory – gauge memory usage and leaks as part your development flow, tools such as ‘memwatch’ can greatly facilitate this task -* Name functions – Minimize the usage of anonymous functions (i.e. inline callback) as a typical memory profiler will provide memory usage per method name -* Use CI tools – Use CI tool to detect failures before sending to production. For example, use ESLint to detect reference errors and undefined variables. Use –trace-sync-io to identify code that uses synchronous APIs (instead of the async version) -* Log wisely – Include in each log statement contextual information, hopefully in JSON format so log aggregators tools such as Elastic can search upon those properties (see separate bullet – ‘Increase visibility using smart logs’). Also, include transaction-id that identifies each request and allows to correlate lines that describe the same transaction (see separate bullet – ‘Include Transaction-ID’) -* Error management – Error handling is the Achilles’ heel of Node.js production sites – many Node processes are crashing because of minor errors while others hang on alive in a faulty state instead of crashing. Setting your error handling strategy is absolutely critical, read here my [error handling best practices](http://goldbergyoni.com/checklist-best-practices-of-node-js-error-handling/) +- The twelve-factor guide – Get familiar with the [Twelve factors](https://12factor.net/) guide +- Be stateless – Save no data locally on a specific web server (see separate bullet – ‘Be Stateless’) +- Cache – Utilize cache heavily, yet never fail because of cache mismatch +- Test memory – gauge memory usage and leaks as part your development flow, tools such as ‘memwatch’ can greatly facilitate this task +- Name functions – Minimize the usage of anonymous functions (i.e. inline callback) as a typical memory profiler will provide memory usage per method name +- Use CI tools – Use CI tool to detect failures before sending to production. For example, use ESLint to detect reference errors and undefined variables. Use –trace-sync-io to identify code that uses synchronous APIs (instead of the async version) +- Log wisely – Include in each log statement contextual information, hopefully in JSON format so log aggregators tools such as Elastic can search upon those properties (see separate bullet – ‘Increase visibility using smart logs’). Also, include transaction-id that identifies each request and allows to correlate lines that describe the same transaction (see separate bullet – ‘Include Transaction-ID’) +- Test like production - Make developers machine quite close to the production infrastructure (e.g., with Docker-Compose). Avoid if/else clauses in testing that check if we're in testing environment but rather run the same code always +- Error management – Error handling is the Achilles’ heel of Node.js production sites – many Node processes are crashing because of minor errors while others hang on alive in a faulty state instead of crashing. Setting your error handling strategy is absolutely critical, read here my [error handling best practices](http://goldbergyoni.com/checklist-best-practices-of-node-js-error-handling/) From 24c207d404fbe62c571ea1cbaa2962ff51b85e28 Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Wed, 10 May 2023 18:30:13 +0300 Subject: [PATCH 4/9] Production practices --- README.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 24d97b520..a64b2daf2 100644 --- a/README.md +++ b/README.md @@ -184,7 +184,7 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [6.24. Prevent unsafe redirects](#-624-prevent-unsafe-redirects)
  [6.25. Avoid publishing secrets to the npm registry](#-625-avoid-publishing-secrets-to-the-npm-registry)
  [6.26. 6.26 Inspect for outdated packages](#-626-inspect-for-outdated-packages)
-  [6.27. Import built-in modules using the 'node:' protocol #new](#-627-import-built-in-modules-using-the-node-protocol)
+  [6.27. Import built-in modules using the 'node:' protocol `#new`](#-627-import-built-in-modules-using-the-node-protocol)
@@ -267,11 +267,11 @@ my-system

-## ![✔] 1.3 Wrap common utilities as npm packages +## ![✔] 1.3 Wrap common utilities as packages, consider publishing -**TL;DR:** In a large app that constitutes a large codebase, cross-cutting-concern utilities like a logger, encryption and alike, should be wrapped by your code and exposed as private npm packages. This allows sharing them among multiple codebases and projects +**TL;DR:** Place all reusable modules in a dedicated folder, e.g., "libraries", and underneath each module in its own folder, e.g., "/libraries/logger". Make the module an independent package with its own package.json file to increases the module encapsulation, and allows future publishing to a repository. In a Monorepo setup, modules can be consumed by 'npm linking' to their physical paths, using ts-paths or by publishing and installing from a package manager repository like the npm registry -**Otherwise:** You'll have to invent your deployment and the dependency wheel +**Otherwise:** Clients of a module might import and get coupled to internal functionality of a module. With a package.json at the root, one can set a package.json.main or package.json.exports to explicitly tell which files and functions are part of the public interface 🔗 [**Read More: Structure by feature**](./sections/projectstructre/wraputilities.md) @@ -279,9 +279,9 @@ my-system ## ![✔] 1.4 Use environment aware, secure and hierarchical config -**TL;DR:** A perfect and flawless configuration setup should ensure (a) keys can be read from file AND from environment variable (b) secrets are kept outside committed code (c) config is hierarchical for easier findability. There are a few packages that can help tick most of those boxes like [rc](https://www.npmjs.com/package/rc), [nconf](https://www.npmjs.com/package/nconf), [config](https://www.npmjs.com/package/config), and [convict](https://www.npmjs.com/package/convict). +**TL;DR:** A flawless configuration setup should ensure (a) keys can be read from file AND from environment variable (b) secrets are kept outside committed code (c) config is hierarchical for easier findability (d) typing support (e) validation for failing fast (f) Specify default for each key. There are a few packages that can help tick most of those boxes like [convict](https://www.npmjs.com/package/convict), [env-var](env-var), [zod](https://github.com/colinhacks/zod), and others -**Otherwise:** Failing to satisfy any of the config requirements will simply bog down the development or DevOps team. Probably both +**Otherwise:** Consider a mandatory environment variable that wasn't provided. The app starts successfully and serve requests, some information is already persisted to DB. Then, it's realized that without this mandatory key the request can't complete, leaving the app in a dirty state 🔗 [**Read More: configuration best practices**](./sections/projectstructre/configguide.md) @@ -798,7 +798,7 @@ b. [Node.js testing - beyond the basics](https://github.com/testjavascript/nodej ## ![✔] 5.1. Monitoring -**TL;DR:** Monitoring is a game of finding out issues before customers do – obviously this should be assigned unprecedented importance. The market is overwhelmed with offers thus consider starting with defining the basic metrics you must follow (my suggestions inside), then go over additional fancy features and choose the solution that ticks all boxes. Click ‘The Gist’ below for an overview of the solutions +**TL;DR:** Monitoring is a game of finding out issues before customers do – obviously this should be assigned unprecedented importance. The market is overwhelmed with offers thus consider starting with defining the basic metrics you must follow (my suggestions inside), then go over additional fancy features and choose the solution that ticks all boxes. In any case, the 4 layers of observability must be covered: uptime, metrics with focus on user-facing symptoms and Node.js technical metrics like event loop lag, distributed flows measurement with Open Telemetry and logging. Click ‘Read More’ below for an overview of the solutions **Otherwise:** Failure === disappointed customers. Simple @@ -1386,9 +1386,11 @@ CMD [ "node", "dist/app.js" ] ## ![✔] 8.2. Bootstrap using `node` command, avoid `npm start` -**TL;DR:** use `CMD ['node','server.js']` to start your app, avoid using npm scripts which don't pass OS signals to the code. This prevents problems with child-processes, signal handling, graceful shutdown and having zombie processes. +**TL;DR:** Use `CMD ['node','server.js']` to start your app, avoid using npm scripts which don't pass OS signals to the code. This prevents problems with child-processes, signal handling, graceful shutdown and having zombie processes -**Otherwise:** When no signals are passed, your code will never be notified about shutdowns. Without this, it will lose its chance to close properly possibly losing current requests and/or data. +Update: [Starting from npm 7, npm claim](https://docs.npmjs.com/cli/v7/using-npm/changelog#706-2020-10-27) to pass signals. We follow and will update accordingly + +**Otherwise:** When no signals are passed, your code will never be notified about shutdowns. Without this, it will lose its chance to close properly possibly losing current requests and/or data [**Read More: Bootstrap container using node command, avoid npm start**](./sections/docker/bootstrap-using-node.md) @@ -1426,7 +1428,7 @@ CMD [ "node", "dist/app.js" ] ## ![✔] 8.6. Shutdown smartly and gracefully -**TL;DR:** Handle the process SIGTERM event and clean-up all existing connection and resources. This should be done while responding to ongoing requests. In Dockerized runtimes shutting down containers is not a rare event, rather a frequent occurrence that happen as part of routine work. Achieving this demands some thoughtful code to orchestrate several moving parts: The load balancer, keep-alive connections, the HTTP server and other resources +**TL;DR:** Handle the process SIGTERM event and clean-up all existing connection and resources. This should be done while responding to ongoing requests. In Dockerized runtimes, shutting down containers is not a rare event, rather a frequent occurrence that happen as part of routine work. Achieving this demands some thoughtful code to orchestrate several moving parts: The load balancer, keep-alive connections, the HTTP server and other resources **Otherwise:** Dying immediately means not responding to thousands of disappointed users From f799d88d26f8dd0d8414be15041eaf5380de7bdd Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Wed, 10 May 2023 18:37:10 +0300 Subject: [PATCH 5/9] Production practices --- README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/README.md b/README.md index a64b2daf2..f9e92e6a4 100644 --- a/README.md +++ b/README.md @@ -271,6 +271,18 @@ my-system **TL;DR:** Place all reusable modules in a dedicated folder, e.g., "libraries", and underneath each module in its own folder, e.g., "/libraries/logger". Make the module an independent package with its own package.json file to increases the module encapsulation, and allows future publishing to a repository. In a Monorepo setup, modules can be consumed by 'npm linking' to their physical paths, using ts-paths or by publishing and installing from a package manager repository like the npm registry +```bash +my-system +├─ apps (components) + │ ├─ component-a +├─ libraries (generic cross-component functionality) +│ ├─ logger +│ │ ├─ package.json +│ │ ├─ src +│ │ │ ├─ index.js + +``` + **Otherwise:** Clients of a module might import and get coupled to internal functionality of a module. With a package.json at the root, one can set a package.json.main or package.json.exports to explicitly tell which files and functions are part of the public interface 🔗 [**Read More: Structure by feature**](./sections/projectstructre/wraputilities.md) From c2af8ff9b28e312c9814495193eafe49694316eb Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Thu, 11 May 2023 10:54:49 +0300 Subject: [PATCH 6/9] Production practices --- README.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index f9e92e6a4..74a17e346 100644 --- a/README.md +++ b/README.md @@ -55,9 +55,9 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin 1. Project Architecture Practices (6) -  [1.1 Structure your solution by components `#strategic`](#-11-structure-your-solution-by-business-components)
-  [1.2 Layer your components, keep the web layer within its boundaries `#strategic`](#-12-layer-your-components-with-3-tiers-keep-the-web-layer-within-its-boundaries)
-  [1.3 Wrap common utilities as npm packages](#-13-wrap-common-utilities-as-npm-packages)
+  [1.1 Structure your solution by components `#strategic` `#updated`](#-11-structure-your-solution-by-business-components)
+  [1.2 Layer your components, keep the web layer within its boundaries `#strategic` `#updated`](#-12-layer-your-components-with-3-tiers-keep-the-web-layer-within-its-boundaries)
+  [1.3 Wrap common utilities as packages, consider publishing](#-13-wrap-common-utilities-as-packages-consider-publishing)
  [1.4 Use environment aware, secure and hierarchical config `#updated`](#-14-use-environment-aware-secure-and-hierarchical-config)
  [1.5 Consider all the consequences when choosing the main framework `#new`](#-15-consider-all-the-consequences-when-choosing-the-main-framework)
  [1.6 Use TypeScript sparingly and thoughtfully `#new`](#-16-use-typescript-sparingly-and-thoughtfully)
@@ -70,15 +70,15 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [2.1 Use Async-Await or promises for async error handling](#-21-use-async-await-or-promises-for-async-error-handling)
-  [2.2 Extend the built-in Error object `#strategic #updated`](#-22-extend-the-built-in-error-object)
+  [2.2 Extend the built-in Error object `#strategic` `#updated`](#-22-extend-the-built-in-error-object)
  [2.3 Distinguish operational vs programmer errors `#strategic` `#updated`](#-23-distinguish-catastrophic-errors-from-operational-errors)
  [2.4 Handle errors centrally, not within a middleware `#strategic`](#-24-handle-errors-centrally-not-within-a-middleware)
-  [2.5 Document API errors using OpenAPI or GraphQL `#updated`](#-25-document-api-errors-using-openapi-or -graphql)
+  [2.5 Document API errors using OpenAPI or GraphQL](#-25-document-api-errors-using-openapi-or -graphql)
  [2.6 Exit the process gracefully when a stranger comes to town `#strategic`](#-26-exit-the-process-gracefully-when-a-stranger-comes-to-town)
  [2.7 Use a mature logger to increase errors visibility `#updated`](#-27-use-a-mature-logger-to-increase-errors-visibility)
-  [2.8 Test error flows using your favorite test framework](#-28-test-error-flows-using-your-favorite-test-framework)
+  [2.8 Test error flows using your favorite test framework `#updated`](#-28-test-error-flows-using-your-favorite-test-framework)
  [2.9 Discover errors and downtime using APM products](#-29-discover-errors-and-downtime-using-apm-products)
-  [2.10 Catch unhandled promise rejections `#modified-recently`](#-210-catch-unhandled-promise-rejections)
+  [2.10 Catch unhandled promise rejections `#updated`](#-210-catch-unhandled-promise-rejections)
  [2.11 Fail fast, validate arguments using a dedicated library](#-211-fail-fast-validate-arguments-using-a-dedicated-library)
  [2.12 Always await promises before returning to avoid a partial stacktrace `#new`](#-212-always-await-promises-before-returning-to-avoid-a-partial-stacktrace)
@@ -90,14 +90,14 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [3.1 Use ESLint `#strategic`](#-31-use-eslint)
-  [3.2 Node.js specific plugins](#-32-nodejs-specific-plugins)
+  [3.2 Use Node.js eslint extension plugins `#updated`](#-32-use-nodejs-eslint-extension-plugins)
  [3.3 Start a Codeblock's Curly Braces on the Same Line](#-33-start-a-codeblocks-curly-braces-on-the-same-line)
  [3.4 Separate your statements properly](#-34-separate-your-statements-properly)
  [3.5 Name your functions](#-35-name-your-functions)
  [3.6 Use naming conventions for variables, constants, functions and classes](#-36-use-naming-conventions-for-variables-constants-functions-and-classes)
  [3.7 Prefer const over let. Ditch the var](#-37-prefer-const-over-let-ditch-the-var)
  [3.8 Require modules first, not inside functions](#-38-require-modules-first-not-inside-functions)
-  [3.9 Require modules by folders, as opposed to the files directly](#-39-require-modules-by-folders-as-opposed-to-the-files-directly)
+  [3.9 Set an explicit entry point to a module/folder `#updated`](#-39-set-an-explicit-entry-point-to-a-modulefolder)
  [3.10 Use the === operator](#-310-use-the--operator)
  [3.11 Use Async Await, avoid callbacks `#strategic`](#-311-use-async-await-avoid-callbacks)
  [3.12 Use arrow function expressions (=>)](#-312-use-arrow-function-expressions-)
@@ -122,7 +122,7 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [4.10 Mock responses of external HTTP services #advanced `#new` `#advanced`](#-410-mock-responses-of-external-http-services)
  [4.11 Test your middlewares in isolation](#-411-test-your-middlewares-in-isolation)
  [4.12 Specify a port in production, randomize in testing `#new`](#-412-specify-a-port-in-production-randomize-in-testing)
-  [4.13 Test the five possible outcomes #strategic #new](#-413-test-the-five-possible-outcomes)
+  [4.13 Test the five possible outcomes #strategic `#new`](#-413-test-the-five-possible-outcomes)
@@ -132,23 +132,23 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [5.1. Monitoring `#strategic`](#-51-monitoring)
-  [5.2. Increase transparency using smart logging `#strategic`](#-52-increase-transparency-using-smart-logging)
+  [5.2. Increase the observability using smart logging `#strategic`](#-52-increase-the-observability-using-smart-logging)
  [5.3. Delegate anything possible (e.g. gzip, SSL) to a reverse proxy `#strategic`](#-53-delegate-anything-possible-eg-gzip-ssl-to-a-reverse-proxy)
  [5.4. Lock dependencies](#-54-lock-dependencies)
  [5.5. Guard process uptime using the right tool](#-55-guard-process-uptime-using-the-right-tool)
  [5.6. Utilize all CPU cores](#-56-utilize-all-cpu-cores)
  [5.7. Create a ‘maintenance endpoint’](#-57-create-a-maintenance-endpoint)
-  [5.8. Discover errors and downtime using APM products `#advanced`](#-58-discover-errors-and-downtime-using-apm-products)
+  [5.8. Discover the unknowns using APM products `#advanced` `#updated`](#-58-discover-the-unknowns-using-apm-products)
  [5.9. Make your code production-ready](#-59-make-your-code-production-ready)
  [5.10. Measure and guard the memory usage `#advanced`](#-510-measure-and-guard-the-memory-usage)
  [5.11. Get your frontend assets out of Node](#-511-get-your-frontend-assets-out-of-node)
-  [5.12. Be stateless, kill your servers almost every day](#-512-be-stateless-kill-your-servers-almost-every-day)
+  [5.12. Strive to be stateless `#strategic`](#-512-strive-to-be-stateless)
  [5.13. Use tools that automatically detect vulnerabilities](#-513-use-tools-that-automatically-detect-vulnerabilities)
  [5.14. Assign a transaction id to each log statement `#advanced`](#-514-assign-a-transaction-id-to-each-log-statement)
  [5.15. Set NODE_ENV=production](#-515-set-node_envproduction)
  [5.16. Design automated, atomic and zero-downtime deployments `#advanced`](#-516-design-automated-atomic-and-zero-downtime-deployments)
  [5.17. Use an LTS release of Node.js](#-517-use-an-lts-release-of-nodejs)
-  [5.18. Don't route logs within the app](#-518-dont-route-logs-within-the-app)
+  [5.18. Log to stdout, avoid specifying log destination within the app](#-518-log-to-stdout-avoid-specifying-log-destination-within-the-app)
  [5.19. Install your packages with npm ci `#new`](#-519-install-your-packages-with-npm-ci)
@@ -209,7 +209,7 @@ Read in a different language: [![CN](./assets/flags/CN.png)**CN**](./README.chin   [8.4. Use .dockerignore to prevent leaking secrets](#-84-use-dockerignore-to-prevent-leaking-secrets)
  [8.5. Clean-up dependencies before production](#-85-clean-up-dependencies-before-production)
  [8.6. Shutdown smartly and gracefully `#advanced`](#-86-shutdown-smartly-and-gracefully)
-  [8.7. Set memory limits using both Docker and v8 `#advanced #strategic`](#-87-set-memory-limits-using-both-docker-and-v8)
+  [8.7. Set memory limits using both Docker and v8 `#advanced` `#strategic`](#-87-set-memory-limits-using-both-docker-and-v8)
  [8.8. Plan for efficient caching](#-88-plan-for-efficient-caching)
  [8.9. Use explicit image reference, avoid latest tag](#-89-use-explicit-image-reference-avoid-latest-tag)
  [8.10. Prefer smaller Docker base images](#-810-prefer-smaller-docker-base-images)
@@ -307,7 +307,7 @@ my-system ## ![✔] 1.6 Use TypeScript sparingly and thoughtfully -**TL;DR:** Coding without type safety is no longer an option, TypeScript is the most popular option for this mission. Use it to define variables and functions return types. With that, it is also a double edge sword that can greatly _encourage_ complexity with its additional ~ 50 keywords and sophisticated features. Consider using it sparingly, mostly with simple types, and utilize advanced features solely when absolutely necessary +**TL;DR:** Coding without type safety is no longer an option, TypeScript is the most popular option for this mission. Use it to define variables and functions return types. With that, it is also a double edge sword that can greatly _encourage_ complexity with its additional ~ 50 keywords and sophisticated features. Consider using it sparingly, mostly with simple types, and utilize advanced features only when a real need arises **Otherwise:** [Researches](https://earlbarr.com/publications/typestudy.pdf) show that using TypeScript can help in detecting ~20% bugs earlier. Without it, also the developer experience in the IDE is intolerable. On the flip side, 80% of other bugs were not discovered using types. Consequently, typed syntax is valuable but limited. Only efficient tests can discover the whole spectrum of bugs, including type-related bugs. It might also defeat its purpose: sophisticated code features are likely to increase the code complexity, which by itself increases both the amount of bugs and the average bug fix time @@ -600,7 +600,7 @@ function doSomething() {

-## ![✔] 3.9 Set an explicit entry point per module/folder +## ![✔] 3.9 Set an explicit entry point to a module/folder **TL;DR:** When developing a module/library, set an explicit root file that exports the public and interesting code. Discourage the client code from importing deep files and becoming familiar with the internal structure. With commonjs (require), this can be done with an index.js file at the folder's root or the package.json.main field. With ESM (import), if a package.json exists on the root, the field "exports" allow specifying the module's root file. If no package.json exists, you may put an index.js file on the root which re-exports all the public functionality From fb8ee242109295786415dbb27a22c8076654f485 Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Thu, 11 May 2023 11:00:33 +0300 Subject: [PATCH 7/9] Production practices --- sections/projectstructre/choose-framework.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/sections/projectstructre/choose-framework.md b/sections/projectstructre/choose-framework.md index 21b32ef0a..da03f6a69 100644 --- a/sections/projectstructre/choose-framework.md +++ b/sections/projectstructre/choose-framework.md @@ -1,4 +1,4 @@ -# Structure your solution by components +# Consider all the consequences when choosing the main framework

@@ -37,8 +37,3 @@ Cons: Covers a small subset of a typical application needs - leaves a handful of **Prefer Fastify when -** The app consists of reasonably-sized components/Microservices (i.e., not a huge monolith); for teams who have solid JavaScript & Node.js knowledge; when sticking to Node.js narratives and spirit is desirable **Prefer Nest.js when** - It's desirable to design and code in OOP style; when the team is highly experienced with Java/Spring/Angular or similar; for large size app that can't be broken down (i.e. monolith) to autonomous component; for a team that lacks fundamental JavaScript/Node.js skills (not exclusively, this yet another consideration); when the decision-making overhead should be minimized; when the time to the first delivery is a critical factor - -

- - -Get lost with express; Nest.js with Fastify, Fastify covers a lot, From 4f9757d212d4aae4c147d8690c28f124ee7f5181 Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Thu, 11 May 2023 11:01:28 +0300 Subject: [PATCH 8/9] Production practices --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 74a17e346..f6c961e01 100644 --- a/README.md +++ b/README.md @@ -297,6 +297,8 @@ my-system 🔗 [**Read More: configuration best practices**](./sections/projectstructre/configguide.md) +

+ ## ![✔] 1.5 Consider all the consequences when choosing the main framework **TL;DR:** When building apps and APIs, using a framework is mandatory. It's easy to overlook alternative frameworks or important considerations and then finally land on a sub optimal option. As of 2023/2024, we believe that these four frameworks are worth considering: [Nest.js](https://nestjs.com/), [Fastify](https://www.fastify.io/), [express](https://expressjs.com/), and [Koa](https://koajs.com/). Click read more below for a detailed pros/cons of each framework. Simplistically, we believe that Nest.js is the best match for teams who wish to go OOP and/or build large-scale apps that can't get partitioned into smaller _autonomous_ components. Fastify is our recommendation for apps with reasonably-sized components (e.g., Microservices) that are built around simple Node.js mechanics. Read our [full considerations guide here](./sections/projectstructre/choose-framework.md) From c8b7eb27aab77ca5c89aa329ebb2b49b249378bf Mon Sep 17 00:00:00 2001 From: Yoni Goldberg Date: Thu, 11 May 2023 11:52:27 +0300 Subject: [PATCH 9/9] Production practices --- README.md | 2 +- sections/projectstructre/choose-framework.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f6c961e01..d811388ee 100644 --- a/README.md +++ b/README.md @@ -305,7 +305,7 @@ my-system **Otherwise:** Due to the overwhelming amount of considerations, it's easy to make decisions based on partial information and compare apples with oranges. For example, it's believed that Fastify is a minimal web-server that should get compared with express only. In reality, it's a rich framework with many official plugins that cover many concerns -🔗 [**Read More: configuration best practices**](./sections/projectstructre/choose-framework.md) +🔗 [**Read More: Choosing the right framework**](./sections/projectstructre/choose-framework.md) ## ![✔] 1.6 Use TypeScript sparingly and thoughtfully diff --git a/sections/projectstructre/choose-framework.md b/sections/projectstructre/choose-framework.md index da03f6a69..6af330a0f 100644 --- a/sections/projectstructre/choose-framework.md +++ b/sections/projectstructre/choose-framework.md @@ -26,7 +26,7 @@ Cons: Younger than others and not as popular yet; smaller eco-system compared to **Koa** -Pros When compared with express: it's Simpler and nimbler; modern API with async/await support; better performance +Pros: When compared with express: it's Simpler and nimbler; modern API with async/await support; better performance Cons: Covers a small subset of a typical application needs - leaves a handful of app concerns uncovered; Not as popular as express and Nest.js