How to (and how not to) maintain your system, GIT and packages

Good admin and his server
Good admin and his server 1000x1000
good_admin_and_his_server_exw.jpg

Let’s consider a standard situation where you have a main work computer on which you have three different projects.
One project is on nodejs, the second is a production project on python, and the third is your personal “pet project”, also on python.
You also have personal and work email in the same system, and, say, a browser and online banking.
And all this under your login.
Well, not under the root login, of course! ¯\_(ツ)_/¯
Everything is quite normal.

Many technically competent developers may have dozens of such projects.
And dozens of keys for SSH or GIT servers.


An example with the popular PyTorch framework


It’s quite ordinary: you write your code, commit it from time to time, and then a torchtriton update arrives in your cozy pet project.
And after that, the following data sets were transferred from your system, in accordance with the binary’s main function:

  • Get system information:
    • nameservers from /etc/resolv.conf
    • hostname from gethostname()
    • current username from getlogin()
    • current working directory name from getcwd()
    • environment variables
  • Read the following files:
    • /etc/hosts
    • /etc/passwd
    • The first 1,000 files in $HOME/*
    • $HOME/.gitconfig
    • $HOME/.ssh/*

The update arrived and the confidential data flew away.
It’s not just everything under your account (and possibly the system) has been compromised, but also, down the chain, everything you managed, committed to, and connected to.

  1. 2023 Compromised PyTorch dependency
  2. 2024 Attack on PyTorch


Second example: 18 npm packages with 2 billion installations per week were compromised


Here’s the most interesting thing, among other things:

This malware is essentially a browser-based interceptor that hijacks both network traffic and application APIs.

  • Injects itself into the browser
    • Hooks core functions like fetch, XMLHttpRequest, and wallet APIs (window.ethereum, Solana, etc.).
    • Ensures it can intercept both web traffic and wallet activity.
  • Watches for sensitive data
    • Scans network responses and transaction payloads for anything that looks like a wallet address or transfer.
    • Recognizes multiple formats across Ethereum, Bitcoin, Solana, Tron, Litecoin, and Bitcoin Cash.
  • Hijacks transactions before they’re signed
    • Alters Ethereum and Solana transaction parameters (e.g., recipients, approvals, allowances).
    • Even if the UI looks correct, the signed transaction routes funds to the attacker.

Your browser is completely compromised! As well as blockchain-based payment instruments.
2FA (two-factor authentication) methods using a phone or a physical token no longer matter - your browser has been patched and is now controlled by malware.

  1. 18 very popular NPM-packages were compromised

I think two examples are enough; there is no need to give and describe more; below I will provide more links to similar attacks.

Code sources


And here’s the most important thing: This code, packages, and their dependencies were distributed from almost trusted sources.
In the first case, it is one of the two most popular frameworks for neural networks, in the second, it is a standard source of NPM packages.


Server side


In addition to your working machine, the code you use is also run after committing when deploying to a server in its own environment and with its own settings, variables, and permissions.
This needs to be understood. Launch environments can vary widely, so I won’t cover them here.


But we need to work with this, right?


At a minimum, you can add a separate user for each project.
For example, your login awesome.
Let’s add a new user ai-pet:

1
2
3
sudo-rs groupadd --gid 12345 ai-pet
sudo-rs adduser --home /home/ai-pet --shell /bin/bash --ingroup ai-pet --disabled-password --uid 12345 ai-pet
sudo-rs usermod -aG ai-pet awesome

sudo-rs visudo
Let’s add a line that allows you to run a specific application (/bin/bash) as another user (ai-pet):
awesome ALL=(ai-pet) /bin/bash
If you don’t want to enter a password, you can add NOPASSWD.
awesome ALL=(ai-pet) NOPASSWD: /bin/bash

Let’s set access parameters for the directory

This is necessary to inherit the rw- permissions for the ai-pet group, of which you are a member.
This will allow you to edit existing and newly created files in this directory.

1
2
3
4
5
sudo-rs mkdir -p /data/git/my-ai-pet
sudo-rs chown ai-pet:ai-pet /data/git/my-ai-pet
sudo-rs chmod u=rwx,g-x,g=rwXs,o-rwx /data/git/my-ai-pet
sudo-rs setfacl -Rm g:ai-pet:rw /data/git/my-ai-pet
sudo-rs setfacl -Rdm g:ai-pet:rw /data/git/my-ai-pet

Or a global setting using umask

Instead of setting parameters for a separate directory, you can configure permissions for created files and directories globally.
sudo-rs nano /home/ai-pet/.bashrc
Let’s add a line:
umask 017

How to use this?

You are awesome. Clone your repository.
cd /data/git/my-ai-pet
git clone ssh://git@development.mysite.net/awesome/ai-project.git

Now log in as ai-pet:
sudo-rs --login --user ai-pet /bin/bash
Now you install all the necessary packages or binaries for this user locally, in ~/.local
If this is python venv then python3 -m venv ~/.my-ai-pet-environment and so on.
If it’s RubyGems, then as an option for .bashrc:

1
2
3
export GEM_HOME=$HOME/.gem
export PATH=$PATH:$GEM_HOME/bin
export PATH=/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin:/home/ai-pet/.gem/bin

Now you can go to your project directory and run the debug server/build or whatever you need.
cd /data/git/my-ai-pet/ai-project

If it’s a fairly simple project that only needs a console to build/run the server, then you can just run commands as ai-pet.
You can edit files in /data/git/my-ai-pet/ai-project and commit changes as awesome.
But just don’t run anything there as awesome!

If this is a complex project that requires running in an IDE, among other things, you can leave the --disabled-password option unset and add a full user, log in as that user, and develop.
At the same time, commit as awesome.

Whatever is installed and run by the user ai-pet, it will have access to everything that is available to its user.
In this case, the ssh key /home/awesome/.ssh/development_mysite_net.ed25519 will remain inaccessible.
And what’s more important, it won’t have access to other keys, like your production one:
/home/awesome/.ssh/enterprise_production_companysite_com.ed25519 and everything else that should not be accessible.
But take into account your “server side”, if this project has one, where this code will also be executed.

What else is important?

File system

The above will make sense if you don’t have directories with important information and permissions like rwxrwxr-x or 775 lurking in your file system due to oversight.
I’ve seen this happen quite often on servers, when a directory was created with a standard mask, then they forgot to change it and put, for example, OpenVPN keys in there.

Net

It’s common to encounter local, unprotected websites with confidential APIs that can be launched for debugging from other projects.
And also local phpmyadmin or pgadmin with passwords like “admin” or “123”.
So after installing the packages, for the duration of launch and debugging, it is quite possible to disable the network for the user ai-pet.
id -u ai-pet

For example:

1
2
3
4
5
6
7
8
iptables -I OUTPUT -o tun0    --match owner --uid-owner 12345 -j DROP
iptables -I OUTPUT -o docker0 --match owner --uid-owner 12345 -j DROP
iptables -I OUTPUT -o kvmbr0  --match owner --uid-owner 12345 -j DROP
iptables -I OUTPUT -o wlan0   --match owner --uid-owner 12345 -j REJECT --reject-with icmp-port-unreachable
iptables -I OUTPUT -o wg0     --match owner --uid-owner 12345 -j DROP
iptables -I OUTPUT -o lo      --match owner --uid-owner 12345 -j ACCEPT
iptables -I OUTPUT -o lo -p udp --dport 53 --match owner --uid-owner 12345 -j REJECT --reject-with icmp-port-unreachable
iptables -I OUTPUT -o lo -p tcp --dport 53 --match owner --uid-owner 12345 -j REJECT --reject-with icmp-port-unreachable


By the way, its network activity can be logged.

But I need it under my login!


And this is normal, for example, I use Wireshark with plugins, and also Kate with additional External Tools.
You must personally read and check all plugins, modules, and scripts that you install.
Instead of running a couple of commands from the README and immediately launching the application:

1
2
3
4
mkdir -p ~/.local/lib/wireshark/plugins/
curl "http://superhackscripts.web/no-more-secrets.lua" > \
    ~/.local/lib/wireshark/plugins/no-more-secrets.lua
wireshark


curl -s | sudo bash


Please don't ever do this!
And here’s why.
Let’s play a game, I’ll call it install cloaking, similar to SEO cloaking.
When one content is displayed to a search bot and another to a user.
Here are three games with slightly different logic:

  • https://secops.it/assets/scripts/install.sh
    In this game, you will be shown one script the first time, then another the next 9 times, and then again.
  • https://secops.it/assets/scripts/install_eo.sh
    In this game you will load alternating scripts, one after the other.
  • https://secops.it/assets/scripts/install_cloaking.sh
    And in this one, you will need a browser and two utilities: curl and wget.

The games are safe, the main thing is not to try to run the code, only read it.
The logic is written quite simple, using the nginx lua module.
In a thought experiment, it could be expanded or narrowed to specific User-Agents of target python package installers or framework update libraries.

This is a clear example that you should read, check and understand the script before running it.
Ideally, checking everything that the installation script downloads, one by one.

1
2
3
curl -v https://somesite.tld/install.sh > install.sh    # Download
less install.sh                                         # Read
sudo bash ./install.sh                                  # Run local copy


Economic feasibility


If you evaluate it superficially, curl | sudo bash will install the required software in a few seconds.
And you are very productive, just lightning fast!
Reading and checking all of its variables is a very time-consuming task.
Installing separate environments and specific sets of packages for each project is also not as quick and convenient.
Maintaining your own private PyPI repository or forks of specific projects is often monstrously time-consuming and expensive.
At the same time, it is unclear what you are doing all the time and why it is necessary if everything is simple for everyone else.

On the other hand, if we ignore the economic, reputational, and informational losses caused by the compromising of just one developer’s workstation, and simply estimate the time required to check everything he had access to, then these are already comparable values.

Some tips


  • Keep ssh/gpg keys encrypted.
  • Don’t forget to set/change access permissions to files and directories.
    Keep an eye on this.
  • Your user, under which you log in, should essentially only manage the environment in which you run absolutely trusted applications if you have any.
  • Keep in mind that any third-party application or script can leak all of your data that it can get its hands on.
  • Any project on GitHub, PyPI, or NPM that you’ve been using for years could update tomorrow and try to kill your system.
    No matter how many millions of stars it has, it depends on its owner, who can at a certain moment:
  • Look towards: KVM/LXC/SElinux/AppArmor


This is unlikely to help against a targeted attack specifically against you, because other vectors may be used, but it is quite suitable against typical mass compromises and worms.

In addition to isolating untrusted, you can similarly isolate trusted environments with minimal sets of packages, dependencies, and data.
KVM is suitable for this purpose, as its system images are easy to transfer between hosts and maintain their settings independently of the hosts.

References


Here are some more notable events in which developers became victims:


I can imagine what would happen if one day there was an update for something like Ansible.