Share this post
Heya, I’m Bruno, and I joined Beamery as a Technical Lead in September 2022. Our team is named Ouroboros – we look after our core data APIs and data quality. Why ‘Ouroboros’? Because everything starts and eventually ends with us. 👀
💡 Could you walk us through a typical day in your role as a Tech Lead at Beamery?
An important part of my role at Beamery is to participate in the sprint and BAU (Business As Usual) of the team. The amount of time spent outside of that often changes with current priorities and workload, but I reckon about two-thirds of my time is usually spent undertaking tasks similar to any other engineer on the team; attending ceremonies, coding (obviously), reviewing code, keeping an eye out for fires, and refining what we need to do next.
And the rest of the time, I’m…
👉 Looking Ahead: Working with our Product Manager, Engineering Manager & stakeholders on scoping out the next “big thing”, ensuring we have detailed enough information and covered the most obvious caveats. For some projects, I may also design a technical solution and/or write an RFC.
👉 Working on our Technical Roadmap: I work with the team’s engineers to assemble our technical roadmap. This covers everything from observability, scalability, security, testing process and developer experience. At Beamery, if you build it, you own it, so we make sure we build it to last and maintain it in the spirit of continuous improvement.
In my role, I am more exposed to what other teams are doing, so I do my best so we can benefit from an economy of scale around knowledge and tooling, and prevent us from reinventing the wheel.
👉 Working on Personal Growth (and that of my colleagues): I participate in maintaining the strong Engineering culture we have across the company. That can be everything from putting together a list of speakers for our monthly Backend Tribe, to using our monthly company-wide Development Day (each month the whole company takes a day to focus on learning and development) to stay on top of the latest trends.
👉 Supporting my team: One of my key responsibilities is to provide mentorship and support to my colleagues. I foster a collaborative learning environment within the team, encouraging the exchange of knowledge and best practices to collectively elevate our skills and deliver outstanding software solutions. This informal yet impactful mentorship plays a vital role in building a more capable and cohesive software engineering team where everyone can thrive and contribute their best to our projects.
⚙️ What are some of the key technologies and tools you use on a daily basis to develop and maintain Beamery’s backend systems?
In our team (Ouroboros), our services are written in Typescript with NestJS. Data is stored in MongoDB and then replicated into ElasticSearch. Some services also use RabbitMQ and Redis. We primarily host on GCP with GKE and use Terraform for IaC.
The tools & services that Ouroboros uses are not mandated across Beamery. Other features or products can utilize Kafka, Python and PostgreSQL.
Our Tech Radar is regularly updated and informs engineering teams of what we can support. Part of being a Tech Lead is reviewing this radar and ensuring we nurture innovation and a best-tool-for-the-job approach while going through the required due diligence.
🧠 In the dynamic field of backend engineering, there must be challenges that arise. Can you share an example of a recent technical challenge you encountered and how you approached solving it?
An example comes to mind of a situation that required close collaboration between multiple delivery teams, our Platform team and our Product Support team. A few months back, we noticed some performance degradation for larger customers on specific API endpoints. We’d recently upgraded MongoDB from version 4 to 5, and the impact correlated with that upgrade.
Our metrics did not show any change in load. Our logs did not show any slow query or increase in the number of queries. This was odd. All dashboards were green, apart from those endpoints that had increased latency.
We had to plug a node debugger into a container and simulate some traffic – on prod-like data (for volumes, we’re talking tens of millions of records), and generate a flame graph. This is when we found it seems most of the time spent was deep in the Mongo native driver, particularly in the toArray method (which is used to serialize the results from multiple cursor reads into an array).
When we upgraded Mongo, we updated the MongoDB Node native driver from 3.6.0 to 4.12.0 (v3 did not support Mongo 5) and made associated code changes to support the new driver.
We discovered the toArray method was significantly refactored in 4.11.0 and it turns out this version was significantly slower than our version of NodeJS; pinning this package to 4.10.0 fixed the issue.
This is a textbook example of smoke hiding the fire. Surely that Mongo upgrade, released just when the issue started, must be the root cause? And if it is a driver issue, surely it comes from the change from v3 to v4?
We tend to trust open-source libraries in the NPM ecosystem that non-major upgrades (like going from 4.10 to 4.11) shouldn’t have adverse effects of this magnitude, and this was a good reminder that the smallest dependency change can have serious consequences!
✨ Outside of core engineering what have you most enjoyed about your time at Beamery?
If you’re like me, and you are too anxious to go to meetups and conferences alone, Beamery goes out of its way to host some pretty interesting ones. In recent months, we have hosted Triangirls, LNUG and Mind the Product! I am thankful those events are brought to me so I don’t need to go to them.
Otherwise, free lunch once a week? Anyone? Preceded (and often succeeded) by a delicious coffee from our amazing in-house barista.
