Blog posts in this series Evaluation of our existing automation tool Evaluation of new automation tools (planned) Inventory migration (planned) Rex Module migration (planned) Rex Wrapper (planned) Testing (planned) Bring it into production (planned) Experiences (planned) Ansible Migration: Evaluation of our existing automation tool In Adjust we are (or maybe have been, by the time you are reading this blog posting) using Rex as an automation tool. It served us well, and we grew the company and the infrastructure based on it....

We recently decided to factor out PostgreSQL authentication into dedicated PgBouncers for different teams, and no longer allow direct remote access to the database cluster. Apart from its main use as a connection pooler, we’ve realized that the isolation afforded by a PgBouncer layer simplifies the management of authentication and connection settings of the different client teams, and frees PostgreSQL of these mundane duties. Motivations In the highly distributed and multi-tenant environment that our databases participate in, even seemingly minor issues easily bloat into an unsustainable mess....

While undertaking our adventure of migrating Rex to Ansible, we came to realize that some of the functionality in Rex wasn’t configuration management, per se. This spawned a more philosophical line of thinking around our Ansible deployment. Let me give you a couple of examples to explain. As Ansible is a configuration tool for servers, it can install software packages using the Operating Systems package management system. Along with installation, it can also do upgrades of software packages on a server....

In our infrastructure, we mostly use Gentoo for our servers. For anyone who doesn’t know Gentoo: this Linux distribution compiles packages on the server. Upgrades take a bit more time, but on the other hand the installation can be adjusted to specific requirements: need debug symbols on a dev or staging platform? No problem, compile them in. Want to optimize an application? Change the compile parameters and recompile once: done. Like with pre-compiled packages, this approach has ups and downs....

In my company, we decided to switch from Rex (an automation tool) to Ansible. It’s a big task, and not knowing Ansible I had to start somewhere. So I looked up some resources for learning. In this article I will describe the different steps you can follow and some tips to start with Ansible. What is Ansible and why use Ansible? First step, I think the best way to learn a new technology like Ansible is to go on the official documentation....

Recently we got alerted by our NetOps team about a replication lag in one PostgreSQL database. That by itself is not a major problem, nevertheless something which deserves investigation. NetOps is using Zabbix, and the replication lag in the database system in question is measured using a plpgsql function provided by an open source template. In the replica, not in the primary. DO LANGUAGE plpgsql $$ DECLARE ver integer; res text; BEGIN SELECT current_setting('server_version_num') INTO ver; IF (ver >= 100000) THEN SELECT * INTO res from ( SELECT CASE WHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN 0 ELSE COALESCE(EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp())::integer, 0) END ) T; ELSE SELECT * INTO res from ( SELECT CASE WHEN pg_last_xlog_receive_location() = pg_last_xlog_replay_location() THEN 0 ELSE COALESCE(EXTRACT(EPOCH FROM now() - pg_last_xact_replay_timestamp())::integer, 0) END ) T; END IF; perform set_config('zbx_tmp....

December 2nd was filled with fun as Adjust Frontend engineers attended React Day Berlin! Here we’d like to capture some of the highlights from the conference. There were many great talks given and we won’t be able to mention every single one of them, but all recordings are available at GitNation portal. Tooling Bun’s builtin JSX transpiler, hot reloads on the server, JSX prop punning, macro API, automatic package installs, console....

The Ansible collection includes the postgresql_privs module which is used to set or revoke privileges on database objects. So far this can be used in a programmatic way: specify the privileges, but also specify if something should be revoked. To give you an example: Let’s say we have a table users in the database. Now an additional role audit is supposed to have SELECT privileges on this table. You solve this by adding one postgresql_privs task:...

This blog here is created using Hugo and is hosted on GitHub Pages. It was part of a Hackathon which Adjust organized in November 2022. The only thing we had to do before the Hackathon was setting up the DNS entry, to give it some time to propagate. andreasscherbaum:~$ dig engineering.adjust.com ; <<>> DiG 9.16.1-Ubuntu <<>> engineering.adjust.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44394 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;engineering....

Nine different teams set out on Monday, Nov 28th, to work on amazing projects. This is the story of the Adjust 2022 Hackathon. Hackathon 2022 Monday morning, 9am, it is early morning and most people are still halfway asleep. The office is empty, but a couple early raisers meet in the ping pong room to hear about the ideas for the Hackathon. Also present are the project leads who spent the last weeks coming up with ideas, and (with thanks to Kat, Zhuo, Peter, Jonas, Viktor, and Pavel) refined these ideas into 2 day challenges....