ci: add treefmt as code formatting multiplexer, refactor CI to avoid duplication, reorg CI into DevOps workflow (#4219)

* ci: add treefmt as code formatting multiplexer

* style: linter findings in entrypoint.sh script

* style: linter findings for yaml and yml

* style: linter findings for json files

* style: linter findings for nix files

* style: linter findings for js files

* style: linter findings for dashboards.sh

* style: linter findings for md and mdx files

* chore: remove unused clang formatter in treefmt config

* style: linter findings for mdx files

* ci: exclude Grafana dashboard JSON files from prettier formatting

* Revert "style: linter findings for json files"

This reverts commit f40c2e1759.

* ci: exclude Grafana dashboard JSON files from all formatting as we use the grafana export style

* style: linter findings for json files

* doc: update changelog

* ci(refactor): use composite action to avoid duplication in elixir workflow

* doc: update changelog

* ci: prevent workflow runs for certain conditions and allow scheduled runs

* ci(refactor): use reusable workflow to check paths

* ci(fix): correct output syntax for check_paths workflow and setting base branch

* ci(refactor): use reusable workflows for streamlined DevOps pipeline

* ci(fix): add write permission for packages in DevOps workflow

* ci(test): test DevOps workflow

* ci(test): test DevOps workflow

* ci(fix): Update condition for spell_check, ensure_linting, elixir, and ghcr_build workflows to reflect empty result instead of false

* ci: revert test DevOps

* ci(refactor): allow ghcr_build parallel to elixir test

* ci(refactor): Remove redundant check_paths job from elixir.yml, elixir_test.yml, and spell_check.yml workflows, check is done in devops.yml

* feat: add treefmt-nix to nix flake (#4219 - @JakobLichterfeld)

* ci: ensure proper linting via treefmt

* ci(test): test ensure_linting workflow

* ci(fix): checkout code for spell_checker to access file to check

* ci(fix): allow impure in ensure_linting workflow

* Revert "ci(test): test ensure_linting workflow"

This reverts commit a67b17ec09.

* ci(fix): correct use of flake-utils for formatter and checks

Co-authored-by: scottbot95 <scottbot95@gmail.com>

* ci(fix): correct use of flake-utils for treefmt

Co-authored-by: scottbot95 <scottbot95@gmail.com>

* refactor: Remove unnecessary imports in flake

* ci(fix): correct syntax in flake

* ci(refactor): Remove unused code in flake.nix

* style: standardised style for input url in flake

* ci(fix): treefmt-nix config with existing options

* ci(feat): Add Nix binary cache and update treefmt command in CI workflow

* ci(refactor): Remove unused code in flake.nix

* fix: include devShell packages only on supported platforms

* fix: update hash for mix-deps package in flake.nix

* ci(fix): Update treefmt command in CI workflow

* ci(test): test ensure_linting workflow

* feat: ensure mix deps are present in devShell

* ci(feat): use flake-parts to enable treefmt-nix

* feat: use flake-parts

* fix: correct use of flake-parts for package build

* doc: update CI badge URL for devops workflow

* ci(fix): handle empty path filter output

* ci: remove --impure flag from treefmt command in CI workflow

* fix: correct treefmt.config settings in formatter.nix

* fix: correct flake-parts inputs, avoid with in imports

* fix: correct program name for mix-format in formatter.nix

* feat: devenv via flake-parts

* fix: correct use of legacy nix code with flake-parts

* ci(fix): correct nix develop command in ensure_linting.yml

* refactor: list imports explicitly in flake, rename folder to flake-modules to be precise

* style: use tabs for indent size to format sh

* style: use nixfmt-rfc-style

* Revert "style: use nixfmt-rfc-style"

This reverts commit 082056159e.

* style: use nixfmt style

* fix: remove glibcLocales from optional dependencies to avoid  "A definition for option `packages."[definition 4-entry 16]"' is not of type `package'."

* fix: remove inotify-tools from optional dependencies to avoid  "A definition for option `packages."[definition 4-entry 16]"' is not of type `package'."

* fix: Remove inotify-tools and glibcLocales from optional dependencie

* fix: correct file paths in flake.nix to version

* fix: add ELIXIR_ERL_OPTIONS to shell environment to force utf8 locale

* fix: add LOCALE_ARCHIVE to shell environment in flake.nix

* Revert "fix: add LOCALE_ARCHIVE to shell environment in flake.nix"

This reverts commit d45f6e32ee.

* ci(refactor): rename workflow to elixir_dep_verification_and_static_analysis.yml to better reflect the intention,, remove duplicate checks

* ci(debug): debug locale settings

* Revert "ci(debug): debug locale settings"

This reverts commit 9b402f3c5f.

* Revert "fix: add ELIXIR_ERL_OPTIONS to shell environment to force utf8 locale"

This reverts commit d02419cba4.

* fix: add LOCALE_ARCHIVE to shell environment in flake.nix

* Revert "fix: add LOCALE_ARCHIVE to shell environment in flake.nix"

This reverts commit 761b437e69.

* fix: add LANG=C.UTF-8 to shell environment in flake.nix

* fix: add mix local.rebar and mix local.hex commands to flake.nix

* fix: pin devenv to version without unix socket bug

* chore: update nixpkgs to nixos-24.05 and update dependencies

* doc: add treefmt config comments

* ci: do not expose treefmt formatter programs in devshell

* fix: correct use of module option to enable PostgreSQL server in flake.nix

* Revert "chore: update nixpkgs to nixos-24.05 and update dependencies"

This reverts commit a6ea3f25aa.

* feat: consistent use of erlang 26 and elixir 1_16 in flake

* ci: switch to macOS runner for linting workflow

* Revert "ci: do not expose treefmt formatter programs in devshell"

This reverts commit 1ecfa451dc.

* Revert "ci: switch to macOS runner for linting workflow"

This reverts commit 7b430667d8.

* ci: Remove nixpkgs channel specification in ensure_linting workflow

* ci(debug): Add debug output for PATH and NIX_PATH in flake.nix

* Revert "ci(debug): Add debug output for PATH and NIX_PATH in flake.nix"

This reverts commit 07faec5ddf.

* fix: avoid the need for impure for devenv

see #4245

* fix: remove invalid custom build.check for formatter and use default

* style: linter findings

* fix: Add emptyTest to avoid nix flake check test execution on non-Linux systems

* chore: Remove LANG=C.UTF-8 from enterShell in flake.nix

* ci(fix): Remove --impure flag from treefmt command in CI mode

* ci(fix): avoid impure mode in ensure_linting workflow

* style: linter findings

* ci(debug): debug elixir version and locale

* chore: Update flake.lock dependencies

* feat: use newer devenv as unix socket bug is fixed in upstream

https://github.com/cachix/devenv/issues/1497

* fix: set rebar3 path in devenv

* Revert "ci(debug): debug elixir version and locale"

This reverts commit 7ecdc77c16.

* ci: re-enable path check in DevOps workflow

* doc: update Development and Contributing guide with nix and treefmt

* ci: use PostgreSQL 17

* style: linter findings

* ci(fix): ensure cache name in build action does not contain invalid characters to avoid invalid reference format

* doc: update changelog

* Revert "ci(fix): ensure cache name in build action does not contain invalid characters to avoid invalid reference format"

This reverts commit 02abb0359d.

* ci: remove branch restriction for check_paths workflow to increase sec

* ci(fix): run ghcr build workflow only for specific conditions

- Ensure workflow only run if there are no changes to the .github folder
- Allow workflow to run on workflow call or PRs from forks
- Prevent duplicate runs for PRs from non-forks
- Avoid invalid reference format for cache name in PRs from our repository

* doc: update changelog

* fix: update hash for mix-deps package in flake.nix

* fix: disable flakeCheck for formatter, as mix format need the dep to be fetched beforehand

* ci(fix): run ghcr build workflow only for specific conditions

* fix: move nixosModules.default to top-level attribute set

* refactor: remove unnecessary config nesting in formatter.nix

* ci(fix): ensure version for buildx is set to correct name

---------

Co-authored-by: scottbot95 <scottbot95@gmail.com>
This commit is contained in:
JakobLichterfeld
2024-10-14 08:54:10 +02:00
committed by GitHub
co-authored by scottbot95
parent ad34131089
commit fdf8185ce2
41 changed files with 1522 additions and 1042 deletions
+250 -185
View File
@@ -2,204 +2,269 @@
description = "TeslaMate Logger";
inputs = {
nixpkgs = { url = "github:NixOS/nixpkgs/nixos-unstable"; };
flake-utils = { url = "github:numtide/flake-utils"; };
devenv.url = "github:cachix/devenv";
devenv-root = {
url = "file+file:///dev/null";
flake = false;
};
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
flake-parts.url = "github:hercules-ci/flake-parts";
flake-parts.inputs.nixpkgs-lib.follows = "nixpkgs";
devenv.url = "github:cachix/devenv/fed89fff44ccbc73f91d69ca326ac241baeb1726"; # https://github.com/cachix/devenv/issues/1497
devenv-root.url = "file+file:///dev/null";
devenv-root.flake = false;
treefmt-nix.url = "github:numtide/treefmt-nix";
treefmt-nix.inputs.nixpkgs.follows = "nixpkgs";
};
outputs = inputs@{ self, nixpkgs, flake-utils, devenv, devenv-root }:
(flake-utils.lib.eachDefaultSystem (system:
let
inherit (pkgs.lib) optional optionals;
pkgs = nixpkgs.legacyPackages.${system};
outputs =
inputs@{ self
, flake-parts
, devenv
, devenv-root
, ...
}:
flake-parts.lib.mkFlake { inherit inputs; } {
elixir = pkgs.beam.packages.erlang.elixir_1_16;
beamPackages = pkgs.beam.packagesWith pkgs.beam.interpreters.erlang;
flake.nixosModules.default = import ./module.nix { inherit self; };
src = ./.;
version = builtins.readFile ./VERSION;
pname = "teslamate";
systems = [
"x86_64-linux"
"aarch64-linux"
"x86_64-darwin"
"aarch64-darwin"
];
# See ./nix/modules/*.nix for the modules that are imported here.
imports = [
inputs.devenv.flakeModule
./nix/flake-modules/formatter.nix
];
mixFodDeps = beamPackages.fetchMixDeps {
TOP_SRC = src;
pname = "${pname}-mix-deps";
inherit src version;
hash = "sha256-ukE7PKrsRaw6TWDSqTJljsPJQedi39mZgGFrMINEJlw=";
# hash = pkgs.lib.fakeHash;
};
perSystem =
{ config
, self'
, inputs'
, pkgs
, system
, ...
}:
# legacy
let
inherit (pkgs.lib) optional optionals;
nixpkgs = inputs.nixpkgs;
pkgs = nixpkgs.legacyPackages.${system};
nodejs = pkgs.nodejs;
nodePackages = pkgs.buildNpmPackage {
name = "teslamate";
src = ./assets;
npmDepsHash = "sha256-05AKPyms4WP8MHBqWMup8VXR3a1tv/f/7jT8c6EpWBw=";
# npmDepsHash = pkgs.lib.fakeHash;
dontNpmBuild = true;
inherit nodejs;
elixir = pkgs.beam.packages.erlang_26.elixir_1_16;
beamPackages = pkgs.beam.packagesWith pkgs.beam.interpreters.erlang_26;
installPhase = ''
mkdir $out
cp -r node_modules $out
ln -s $out/node_modules/.bin $out/bin
src = ./.;
version = builtins.readFile ./VERSION;
pname = "teslamate";
rm $out/node_modules/phoenix
ln -s ${mixFodDeps}/phoenix $out/node_modules
mixFodDeps = beamPackages.fetchMixDeps {
TOP_SRC = src;
pname = "${pname}-mix-deps";
inherit src version;
hash = "sha256-Y+CGgvnSCiiuyhtsQ+j0vayq1IHO5IEPVl+V/wwTd6w=";
# hash = pkgs.lib.fakeHash;
};
rm $out/node_modules/phoenix_html
ln -s ${mixFodDeps}/phoenix_html $out/node_modules
nodejs = pkgs.nodejs;
nodePackages = pkgs.buildNpmPackage {
name = "teslamate";
src = ./assets;
npmDepsHash = "sha256-05AKPyms4WP8MHBqWMup8VXR3a1tv/f/7jT8c6EpWBw=";
# npmDepsHash = pkgs.lib.fakeHash;
dontNpmBuild = true;
inherit nodejs;
rm $out/node_modules/phoenix_live_view
ln -s ${mixFodDeps}/phoenix_live_view $out/node_modules
installPhase = ''
mkdir $out
cp -r node_modules $out
ln -s $out/node_modules/.bin $out/bin
rm $out/node_modules/phoenix
ln -s ${mixFodDeps}/phoenix $out/node_modules
rm $out/node_modules/phoenix_html
ln -s ${mixFodDeps}/phoenix_html $out/node_modules
rm $out/node_modules/phoenix_live_view
ln -s ${mixFodDeps}/phoenix_live_view $out/node_modules
'';
};
cldr = pkgs.fetchFromGitHub {
owner = "elixir-cldr";
repo = "cldr";
rev = "v2.37.5";
sha256 = "sha256-T5Qvuo+xPwpgBsqHNZYnTCA4loToeBn1LKTMsDcCdYs=";
# sha256 = pkgs.lib.fakeHash;
};
pkg = beamPackages.mixRelease {
TOP_SRC = src;
inherit
pname
version
elixir
src
mixFodDeps
;
LOCALES = "${cldr}/priv/cldr";
postBuild = ''
ln -sf ${mixFodDeps}/deps deps
ln -sf ${nodePackages}/node_modules assets/node_modules
export PATH="${pkgs.nodejs}/bin:${nodePackages}/bin:$PATH"
${nodejs}/bin/npm run deploy --prefix ./assets
# for external task you need a workaround for the no deps check flag
# https://github.com/phoenixframework/phoenix/issues/2690
mix do deps.loadpaths --no-deps-check, phx.digest
mix phx.digest --no-deps-check
'';
meta = {
mainProgram = "teslamate";
};
};
postgres_port = 7000;
mosquitto_port = 7001;
process_compose_port = 7002;
psql = pkgs.writeShellScriptBin "teslamate_psql" ''
exec "${pkgs.postgresql}/bin/psql" --host "$DATABASE_HOST" --user "$DATABASE_USER" --port "$DATABASE_PORT" "$DATABASE_NAME" "$@"
'';
};
cldr = pkgs.fetchFromGitHub {
owner = "elixir-cldr";
repo = "cldr";
rev = "v2.37.5";
sha256 = "sha256-T5Qvuo+xPwpgBsqHNZYnTCA4loToeBn1LKTMsDcCdYs=";
# sha256 = pkgs.lib.fakeHash;
};
pkg = beamPackages.mixRelease {
TOP_SRC = src;
inherit pname version elixir src mixFodDeps;
LOCALES = "${cldr}/priv/cldr";
postBuild = ''
ln -sf ${mixFodDeps}/deps deps
ln -sf ${nodePackages}/node_modules assets/node_modules
export PATH="${pkgs.nodejs}/bin:${nodePackages}/bin:$PATH"
${nodejs}/bin/npm run deploy --prefix ./assets
# for external task you need a workaround for the no deps check flag
# https://github.com/phoenixframework/phoenix/issues/2690
mix do deps.loadpaths --no-deps-check, phx.digest
mix phx.digest --no-deps-check
mosquitto_sub = pkgs.writeShellScriptBin "teslamate_sub" ''
exec "${pkgs.mosquitto}/bin/mosquitto_sub" -h "$MQTT_HOST" -p "$MQTT_PORT" -u "$MQTT_USERNAME" -P "$MQTT_PASSWORD" "$@"
'';
meta = {
mainProgram = "teslamate";
devShell = inputs.devenv.lib.mkShell {
inherit inputs pkgs;
modules = with pkgs; [
{
devenv.root =
let
devenvRootFileContent = builtins.readFile devenv-root.outPath;
in
pkgs.lib.mkIf (devenvRootFileContent != "") devenvRootFileContent;
packages =
[
elixir
elixir_ls
node2nix
nodejs
prefetch-npm-deps
# for dashboard scripts
jq
psql
mosquitto
mosquitto_sub
config.treefmt.build.wrapper
]
++ builtins.attrValues config.treefmt.build.programs
# ++ optional stdenv.isLinux [
# inotify-tools # disabled to avoid error: A definition for option `packages."[definition 4-entry 16]"' is not of type `package'.
# glibcLocales # disabled to avoid error: A definition for option `packages."[definition 4-entry 16]"' is not of type `package'.
# ]
++ optional stdenv.isDarwin terminal-notifier
++ optionals stdenv.isDarwin (
with darwin.apple_sdk.frameworks;
[
CoreFoundation
CoreServices
]
);
enterShell = ''
export LOCALES="${cldr}/priv/cldr";
export PORT="4000"
export ENCRYPTION_KEY="your_secure_encryption_key_here"
export DATABASE_USER="teslamate"
export DATABASE_PASS="your_secure_password_here"
export DATABASE_NAME="teslamate"
export DATABASE_HOST="127.0.0.1"
export DATABASE_PORT="${toString postgres_port}"
export MQTT_HOST="127.0.0.1"
export MQTT_PORT="${toString mosquitto_port}"
export RELEASE_COOKIE="1234567890123456789"
export TZDATA_DIR="$PWD/tzdata"
export MIX_REBAR3="${rebar3}/bin/rebar3";
mix deps.get
'';
enterTest = ''
mix test
'';
processes.mqtt = {
exec = "${pkgs.mosquitto}/bin/mosquitto -p ${toString mosquitto_port}";
};
process.managers.process-compose = {
port = process_compose_port;
tui.enable = true;
};
services.postgres = {
enable = true;
package = pkgs.postgresql_16; # 17 is not yet available in nixpkgs
listen_addresses = "127.0.0.1";
port = postgres_port;
initialDatabases = [{ name = "teslamate"; }];
initialScript = ''
CREATE USER teslamate with encrypted password 'your_secure_password_here';
GRANT ALL PRIVILEGES ON DATABASE teslamate TO teslamate;
ALTER USER teslamate WITH SUPERUSER;
'';
};
}
];
};
moduleTest =
(nixpkgs.lib.nixos.runTest {
hostPkgs = pkgs;
defaults.documentation.enable = false;
imports = [
{
name = "teslamate";
nodes.server = {
imports = [ self'.nixosModules.default ];
services.teslamate = {
enable = true;
secretsFile = builtins.toFile "teslamate.env" ''
ENCRYPTION_KEY=123456789
DATABASE_PASS=123456789
RELEASE_COOKIE=123456789
'';
postgres.enable_server = true;
grafana.enable = true;
};
};
testScript = ''
server.wait_for_open_port(4000)
'';
}
];
}).config.result;
emptyTest = pkgs.stdenv.mkDerivation {
name = "noTest";
buildPhase = ''
echo "Tests are only supported on Linux."
'';
};
in
{
packages = {
devenv-up = devShell.config.procfileScript;
default = pkg;
};
devShells.default = devShell;
# for `nix flake check`
checks = {
default = if pkgs.stdenv.isLinux then moduleTest else emptyTest;
# formatter check is done in the formatter module
};
};
postgres_port = 7000;
mosquitto_port = 7001;
process_compose_port = 7002;
psql = pkgs.writeShellScriptBin "teslamate_psql" ''
exec "${pkgs.postgresql}/bin/psql" --host "$DATABASE_HOST" --user "$DATABASE_USER" --port "$DATABASE_PORT" "$DATABASE_NAME" "$@"
'';
mosquitto_sub = pkgs.writeShellScriptBin "teslamate_sub" ''
exec "${pkgs.mosquitto}/bin/mosquitto_sub" -h "$MQTT_HOST" -p "$MQTT_PORT" -u "$MQTT_USERNAME" -P "$MQTT_PASSWORD" "$@"
'';
devShell = devenv.lib.mkShell {
inherit inputs pkgs;
modules = with pkgs; [{
devenv.root =
let
devenvRootFileContent = builtins.readFile devenv-root.outPath;
in
pkgs.lib.mkIf (devenvRootFileContent != "") devenvRootFileContent;
packages = [
elixir
elixir_ls
glibcLocales
node2nix
nodejs
prefetch-npm-deps
# for dashboard scripts
jq
psql
mosquitto
mosquitto_sub
] ++ optional stdenv.isLinux inotify-tools
++ optional stdenv.isDarwin terminal-notifier
++ optionals stdenv.isDarwin (with darwin.apple_sdk.frameworks; [
CoreFoundation
CoreServices
]);
enterShell = ''
export LOCALES="${cldr}/priv/cldr";
export PORT="4000"
export ENCRYPTION_KEY="your_secure_encryption_key_here"
export DATABASE_USER="teslamate"
export DATABASE_PASS="your_secure_password_here"
export DATABASE_NAME="teslamate"
export DATABASE_HOST="127.0.0.1"
export DATABASE_PORT="${toString postgres_port}"
export MQTT_HOST="127.0.0.1"
export MQTT_PORT="${toString mosquitto_port}"
export RELEASE_COOKIE="1234567890123456789"
export TZDATA_DIR="$PWD/tzdata"
'';
enterTest = ''
mix test
'';
processes.mqtt = {
exec =
"${pkgs.mosquitto}/bin/mosquitto -p ${toString mosquitto_port}";
};
process.process-compose = {
port = process_compose_port;
tui = true;
};
services.postgres = {
enable = true;
package = pkgs.postgresql_16;
listen_addresses = "127.0.0.1";
port = postgres_port;
initialDatabases = [{ name = "teslamate"; }];
initialScript = ''
CREATE USER teslamate with encrypted password 'your_secure_password_here';
GRANT ALL PRIVILEGES ON DATABASE teslamate TO teslamate;
ALTER USER teslamate WITH SUPERUSER;
'';
};
}];
};
moduleTest = (nixpkgs.lib.nixos.runTest {
hostPkgs = pkgs;
defaults.documentation.enable = false;
imports = [{
name = "teslamate";
nodes.server = {
imports = [ self.nixosModules.default ];
services.teslamate = {
enable = true;
secretsFile = builtins.toFile "teslamate.env" ''
ENCRYPTION_KEY=123456789
DATABASE_PASS=123456789
RELEASE_COOKIE=123456789
'';
postgres.enable = true;
grafana.enable = true;
};
};
testScript = ''
server.wait_for_open_port(4000)
'';
}];
}).config.result;
in {
packages = {
devenv-up = devShell.config.procfileScript;
default = pkg;
};
devShells.default = devShell;
checks.default = moduleTest;
})) // {
nixosModules.default = import ./module.nix { inherit self; };
};
};
}