On a lovely Saturday afternoon our NetOps team raised an alarm: the disk usage on a specific database server was growing fast, and with the current growth rate we have about 30 hours left until the disk is full. Currently climbing over 80%, which triggered the alarm in the first place. Not good. However nothing serious as of now, but better investigate now and not wait until Monday. Zabbix Graph: Space utilization The System This particular server has about 3....
Introduction In Adjust we are creating new data types for PostgreSQL, in order to make handling of data easier for us, but also sometimes to save some space. Many of them are written using the C language. Here are some of the more important extensions for us: istore: it is an integer based hstore. In istore both keys and values are represented and stored as integers. pg-base36: a base36 extension, which implements a base36 binary-to-text encoding algorithm....
Blog posts in this series Evaluation of our existing automation tool Evaluation of new automation tools (planned) Inventory migration (planned) Rex Module migration (planned) Rex Wrapper (planned) Testing (planned) Bring it into production (planned) Experiences (planned) Ansible Migration: Evaluation of our existing automation tool In Adjust we are (or maybe have been, by the time you are reading this blog posting) using Rex as an automation tool. It served us well, and we grew the company and the infrastructure based on it....
We recently decided to factor out PostgreSQL authentication into dedicated PgBouncers for different teams, and no longer allow direct remote access to the database cluster. Apart from its main use as a connection pooler, we’ve realized that the isolation afforded by a PgBouncer layer simplifies the management of authentication and connection settings of the different client teams, and frees PostgreSQL of these mundane duties. Motivations In the highly distributed and multi-tenant environment that our databases participate in, even seemingly minor issues easily bloat into an unsustainable mess....
While undertaking our adventure of migrating Rex to Ansible, we came to realize that some of the functionality in Rex wasn’t configuration management, per se. This spawned a more philosophical line of thinking around our Ansible deployment. Let me give you a couple of examples to explain. As Ansible is a configuration tool for servers, it can install software packages using the Operating Systems package management system. Along with installation, it can also do upgrades of software packages on a server....
In our infrastructure, we mostly use Gentoo for our servers. For anyone who doesn’t know Gentoo: this Linux distribution compiles packages on the server. Upgrades take a bit more time, but on the other hand the installation can be adjusted to specific requirements: need debug symbols on a dev or staging platform? No problem, compile them in. Want to optimize an application? Change the compile parameters and recompile once: done. Like with pre-compiled packages, this approach has ups and downs....
In my company, we decided to switch from Rex (an automation tool) to Ansible. It’s a big task, and not knowing Ansible I had to start somewhere. So I looked up some resources for learning. In this article I will describe the different steps you can follow and some tips to start with Ansible. What is Ansible and why use Ansible? First step, I think the best way to learn a new technology like Ansible is to go on the official documentation....
Recently we got alerted by our NetOps team about a replication lag in one PostgreSQL database. That by itself is not a major problem, nevertheless something which deserves investigation. NetOps is using Zabbix, and the replication lag in the database system in question is measured using a plpgsql function provided by an open source template. In the replica, not in the primary. DO LANGUAGE plpgsql $$ DECLARE ver integer; res text; BEGIN SELECT current_setting('server_version_num') INTO ver; IF (ver >= 100000) THEN SELECT * INTO res from ( SELECT CASE WHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN 0 ELSE COALESCE(EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp())::integer, 0) END ) T; ELSE SELECT * INTO res from ( SELECT CASE WHEN pg_last_xlog_receive_location() = pg_last_xlog_replay_location() THEN 0 ELSE COALESCE(EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp())::integer, 0) END ) T; END IF; perform set_config('zbx_tmp....
December 2nd was filled with fun as Adjust Frontend engineers attended React Day Berlin! Here we’d like to capture some of the highlights from the conference. There were many great talks given and we won’t be able to mention every single one of them, but all recordings are available at GitNation portal. Tooling Bun’s builtin JSX transpiler, hot reloads on the server, JSX prop punning, macro API, automatic package installs, console....
The Ansible collection includes the postgresql_privs module which is used to set or revoke privileges on database objects. So far this can be used in a programmatic way: specify the privileges, but also specify if something should be revoked. To give you an example: Let’s say we have a table users in the database. Now an additional role audit is supposed to have SELECT privileges on this table. You solve this by adding one postgresql_privs task:...