Skip to main content

Dependency management

With so many services that are required to run in parallel in the same system, there will inevitably be conflicts in dependencies, be it PIP packages (for microservices written in Python) or npm modules (for the different systems written in Node.js). Cue, package managers.

Python package manager, Poetry​

Poetry is our Python packaging and dependency management tool of choice. Do familiarise yourself with the basic usage of Poetry, as well as where the virtual environments of Poetry are stored and how to use these virtual environments in your daily scripting. Notably, read up on how Poetry resolves dependencies by splitting them into dependency groups.

You should be familiar with the following Poetry concepts:

  • poetry install
  • poetry add
  • poetry remove
  • poetry update
  • poetry run
  • The pyproject.toml and poetry.lock files
  • Some command Poetry configurations such as cache-dir

Node.js package manager, pnpm​

For Node.js packages, we use pnpm to manage dependencies. To fully appreciate pnpm, first familiarise yourself with npm, the default node package manager that comes pre-installed when installing Node.js.

Some of the common npm commands to be familiar with:

  • npm install (package vs global installation)
  • npm ci (clean install)
  • npm update
  • npm uninstall
  • npm run and the scripts in the package.json file

The main reason why we migrated to pnpm is due to the continuous re-downloading of packages, which are extremely common in Node.js development workflows. Working behind the proxy is especially painful, as we use packages that can get sufficiently large in size such as the Cypress binary. pnpm works on a symlink basis, helping to save disk space and boosting installation speed by reducing network calls to the npm registry. Read more about the speed improvements here.

However, due to the slight difference in structure in npm vs pnpm, there are a few key differences in configurations, especially when it comes to monorepo setups.

What is a monorepo?

A monorepo is a version-controlled code repository that holds multiple projects in the same repository. For instance, ACUBETotal is made up of multiple projects - frontend, authorization, pipeline, and numerous Python microservices.

Some key benefits of using a monorepo include coordinating of commits across multiple projects, and much more code reuse, allowing the monorepo to act as a "single source of truth".

Some key differences in pnpm that you may need to be aware of:

  • pnpm-workspaces.yaml file and its significance
  • .npmrc
  • workspace dependencies in package.json

Workspaces​

As we work in a monorepo, you might need to be familiar with workspaces. For instance, adding a package into the authorization workspace will not make that package available in other workspaces.

Similarly, for python packages, adding a dependency in a poetry project does not make that dependency available in other poetry projects.

tip

Ensure that the correct environment is used when handling multiple projects with different environments. For example, when dealing with multiple Poetry projects, ensure that your Python environment that VSCode is using is the appropriate one. A quick and easy way to double check is to run poetry install and check the path of the virtualenv created by Poetry, and point VSCode to that specific environment. This way, any unavailable packages will be flagged out immediately by IntelliSense.