pablohere/public/writings/thoughts-after-departing-galoy.html

148 lines
9.6 KiB
HTML

<!DOCTYPE html>
<html>
<head>
<title>Pablo here</title>
<meta charset="utf-8" />
<meta viewport="width=device-width, initial-scale=1" />
<link rel="stylesheet" href="../styles.css" />
</head>
<body>
<main>
<h1>Hi, Pablo here</h1>
<p><a href="../index.html">back to home</a></p>
<hr />
<section>
<h2>Thoughts after departing Galoy</h2>
<p><em>Published: 2026-08-30</em></p>
<p>
I recently left my position as a Data Engineer at Galoy to pursue some other opportunities. I thought it would
be a good time to reflect a bit and jot down some notes for my future self. I did that for the first time when
leaving Superhog and it proved useful, so here it goes.
</p>
<ul>
<li>
Having a technical CEO saves you from many pains that a non-technical CEO will bring to your door. But it
will introduce other ones. It is definitely not heaven on earth compared to non-technical CEOs, but rather a
different set of trade-offs. If you care about being surrounded by a culture of great engineering, though, a
technical CEO really helps.
</li>
<li>
The team stayed away from anything that resembled a production environment for my full tenure. I could see how
it rots away the spirit. There are issues with having a bunch of engineers crank code against a future
hypothetical usage that never comes. Eventually, care degrades. You do not build things the same way when the
quality bar is being able to call the task done, even with very good CI and a good culture of testing, as you
do when your stuff will actually be used in the real world. Operations also take a hit: when you can always
reset any environment, because you do not live under the terrible threat of at least one true production
environment that needs to be treated like a jewel, you tolerate many deployment and operation quirks. Finally,
without clear guidance provided by business needs, the team will sometimes deviate into funky technical nerd
snipes that, after weeks or months, might end up providing nothing useful to the product or business.
</li>
<li>
Working with a globally distributed remote team can be a challenge. You can get the feeling that things slow
down dramatically because communication sometimes ends up working with one working day of latency: you ask
your coworker, who is sleeping, something; then he replies back when you are AFK. Now I can see why some
companies make it strict to only hire people within certain timezone boundaries. Also, executing an offsite is
a logistics nightmare. No matter which place you pick to meet, someone will have visa nightmares to get there.
</li>
<li>
It is insanely productive to have a strong CI setup and to heavily automate all things related to builds,
testing, deployment, and so on. I was a bit hesitant about going all the way with certain automations, but this
team really showed me tricks and patterns to take CI to the limit and I loved it.
</li>
<li>
Compiling Rust is a pain in the ass.
</li>
<li>
I have seen many places where you always have one engineer owning one task, and the idea of putting two people
under the same task would be considered unproductive. In Galoy, it was quite the opposite: having multiple
engineers on any non-trivial task was considered to be faster in overall team velocity, because multiple
people being hands-on with the same task would speed up design and review. I fell in love with the concept,
and now I find a lot of value in mobbing. It is also great for knowledge sharing. Galoy implemented some rules
to motivate this, like limiting PRs in flight and having the convention for engineers to always try to join a
WIP task instead of getting a brand new one started.
</li>
<li>
Incident reviews were always done collectively: whoever might have tackled the incident started a few notes,
but there were large chunks of the report that would be filled in collectively before the review meeting. Then
many details would be added during the meeting itself. The bit I found most interesting about this shared
ownership of the review was that, during the meeting, each related engineer was expected to walk everyone else
through their own subjective timeline of events, uninterrupted. This practice, as opposed to someone playing
detective and trying to craft one global sequence of events, was really helpful in making evident the
misalignments in how different colleagues collaborated together.
</li>
<li>
If you are the kind of person who likes to get shit done, traditional banking is a terrible place to be. It is
the most regulated, risk-averse environment I have ever lived in. Innovating there is almost impossible. This
is the reason you see fintechs do cool stuff and banks not: the fintech is working outside of financial
regulations, and only touching the industry in the most minimal way possible.
</li>
<li>
Another related thought: running a tradfi bank is so utterly complex due to regulation that bootstrapping one
from scratch somehow feels impossible. You are going to need double-digit millions just to get ready for
licensing and launch, with no guarantee that you will actually take off successfully. It feels like a system
made to ensure no new players enter, which is obviously in the interest of both current players and regulators.
One specifically tough part here is that licensing is a bit all or nothing: if you want to start the bank, you
must prove to the regulator that you can properly operate up to his standards everything the license offers,
even if you only intend to offer a very limited set of products. You cannot get an MVP license: you are either
a bank or not, and you cannot lean-launch yourself into existence. It reminds me of the famous John Gall quote:
<em>A complex system that works is invariably found to have evolved from a simple system that worked.</em>
Here, regulators have doomed startups by forcing everyone to somehow miraculously start being complex out the
door.
</li>
<li>
I got the chance to visit El Salvador. It is a really poor country, but at the time of my visit the spirits
were high because things had been trending upwards for years already. The environment created by such a winning
streak was awesome. I found many people were very happy because, even if they were still in deep shit, things
had been improving significantly for them for quite some time, and they expected more good things to come.
</li>
<li>
I have doubted in the past about whether scheduled 1:1s, retrospectives, and performance reviews are good or
not. The good old debate between "schedule it and do it with some frequency" versus "let us just raise the
conversation when needed". After being in Galoy, I have finally come to the personal conclusion that it pays
off to have those meetings scheduled on a recurrent basis. If you feel you have nothing to say, just schedule
them less frequently.
</li>
<li>
Event sourcing is a beautiful pattern. It comes with its own complexity, but if you are in a system with true
playback and audit needs, nothing beats it. The only time I found it can give you a headache is when you deal
with requests to delete sensitive personal data.
</li>
<li>
As you try to scale throughput, strong consistency increasingly becomes an obstacle. I now understand why so
many other systems I dealt with in the past force you through async "fill a request and poll the state"
patterns: they are shoving you into an async background queue to take weight away from the overloaded main
server.
</li>
<li>
Having all the tools you usually interact with when working, both development and operations related, exposing a
CLI is awesome. The ability to mix and match information from different spots in a single terminal is very
productive. Both as a human, but also when working with LLM agents.
</li>
<li>
Testing with BATS is great. I now realize there are so many other projects that could have had great testing
if they had stepped out one more layer of the onion and run their tests at the terminal layer.
</li>
<li>
Having things that you can check in CI but not locally inevitably leads to headaches. It pays off to make all
the CI tasks runnable locally. Instead of relying on complicated multistep workflows in the CI YAMLs, perhaps
just try to have one bash script per task that runs the same locally and in the CI VM.
</li>
<li>
If you are trying to sell critical products or services to big corps, in the sense that their business will
strongly depend on what you are providing, having weak financials is a serious liability. They may love your
proposal, but they may also get really scared from the fact that it looks like you are going bankrupt in a few
months.
</li>
</ul>
<hr />
<p><a href="../index.html">back to home</a></p>
</section>
</main>
</body>
</html>