Building engineering tools as a group

Navaneesh Kumar

Mar 6, 2022 • 4 min read

Great engineering tools boost the forging of high-quality technology products. As engineers, we enjoy using various tools that enhance the productivity of our day-to-day operations. We rejoice in the power yielded to us for building technology - be it for ourselves or the world.

Curiosity is one of the three virtues at Hexmos, with Consistency and Audacity being the other two. Developing useful internal engineering tools keeps a good amount of control within the organization, improves developer productivity, reduces external dependencies, enables collaboration, and brings a unique flavor to Hexmos. Internal tools are the best playground for engineers. They offer a low-risk environment for experimenting with technology (break things, be audacious) and provide an easy transition to newcomers.

We strive to maintain a good balance between “doing work” and “building tools.” We use our common sense to make decisions (yes, it is ambiguous, so is the reality). Keeping org members' long-term interest and curiosity overrides short-term losses in product building. We invest our time and energy in building tools that facilitate building significant products.

Some examples from the industry

Google develops its products and features with the right balance of speed and quality. On the one hand, the credit goes to world-class engineers. Still, the crucial part of their success is investing in developer infrastructure and tools.
Amazon developed its tools organically for internal consumption for a decade - EC2, for example. AWS started by evolving EC2 and other internal tools like Simple Queue Service (SQS) and Simple Storage Service (S3).
Donald Knuth took one year off from publishing as a mathematician as he was unhappy with the tool for publishing. He reasoned - “it would be better to spend a year writing new tools than to try to get good results out of the old one.” He built TeX in 1978, a tool almost every researcher uses today, even after 45 years.
Guido van Rossum created the Python programming language and evolved it as a tool. Guido mentions his motivation for creating the language in his blog - “I realized that the development of system administration utilities in C was taking too long.”
Unified Payment Interface (UPI) is powerful for its 3-layer architecture enabling developers to build financial applications involving money transactions. The objective is super straightforward - “solution should be scalable to a billion users and large scale adoption.”
IKEA sells “ready-to-assemble” furniture with physical stores in more than 60 countries. If you buy a chair from IKEA, you need to assemble it yourself. The steps for assembly are adequately complex and require your brain for 30-60min. IKEA will include appropriate tools (Allen key, screws, etc.) with the purchase. They used the psychology of people's desire to “assemble” themselves, famously known as the IKEA effect.

Tool building to accelerate bootstrapping

We develop tools in the context of our group, not just individuals. Try to answer the following:

Can you increase your team's productivity or even the entire organization?
Does the tool enable collaboration? Can someone else contribute and improve the features quickly?
Can you get 100 people to use it?

It is relatively easy to build tools that satisfy your personal needs. Still, it is challenging to get it right for a larger audience. However, powerful tools can enable bootstrapping, solve problems as a group, and impact the whole organization.

“If you tools are faulty, then unless you can tell, you will often just blame yourself” - Brandon Rhodes

For example, we adopted Impression River for its usability throughout our sessions. But, due to coupled internal structure, none of us could contribute features other than the original authors. On the contrary, woof rapidly enabled collaboration among ourselves. The project structure made adding new functions relatively more straightforward and organic.

Hexmos engineering tools

SSLify

We built SSLify for automating creating HTTPS endpoints for our services. Focusing on simplicity and usability, you can convert an HTTP service endpoint to HTTPS with a single command:

python sslify.py --url newapp.apps.hexmos.com --service-ip 65.1.38.25 --service-port 8932

A single command takes care of many manual steps. So far, we have enabled HTTPS for ten services using SSLify. Without SSLify, we would end up with heterogeneous nginx configurations, waste of time and energy (multiplied for each service), severe impact to productivity, and run into frustration.

httpelf

API testing is an essential part of service development. We struggled with the bloated interface of Postman and the unusability of Insomnia. Our requirements are simple, which both of the tools didn't offer easily:

Collaborate and persist API specs by simple Git integration
A lightweight and straightforward interface to call API endpoints.

Httpelf uses a custom format for writing your API specs (URL, body, method, etc.). You can call the API with a single command:

elf my_endpoint.http

Effectively, all Hexmos service APIs are located in the apihub repository organized by projects. We can test any endpoint with one single command.

woof

Woof (Write Outrageously Omnipotent Functions) is inspired by Facebook's internal tool - bunny. Anyone in the organization can write a simple Python function and make it available to everybody. Even though the idea sounds simple, within 45 days, we have got 6 handy functions:

gls - search across all of Hexmos Gitlab repositories
wks - search Hexmos Gitlab wiki
ir - lists team members in each room in Impression River.
phpbb - visualizes consistency of Hexmos members in the daily reading program
sesgen - Generates Amazon Simple Email Service (SES) configurations in one step (reducing error-prone manual steps)
z - Fuzzy jump to any of our Gitlab repositories.

With Woof, we quickly onboarded our new team members, acting as a powerful catalyst for getting started with the technology stack.

Our Open source libraries

Python AWS SQS consumer - We will be using a lot of asynchronous patterns in our future system architectures. With this library, we can write Amazon Simple Queue Service (SQS) consumers in Python with a simplified interface.
Metric Gateway - Observability is a key component of any service. We ingest metrics for better visibility in Impression River via Prometheus. This library should evolve and act as a single metric ingestion point for all of our services in future.
R2D2 RedisGraph (Rust)

Our philosophies and learnings

We slow down in the short term to speed up the long term. If any programming work is getting monotonous, stop, and automate it. Even better - automate for everyone.
Be mentally prepared to scrap your tools as your environment and context change.
Products can emerge out of tools, organically, just like nature's evolution. It is internally battle-tested, validated, and better than starting from a whiteboard.
Look for adoption early, get feedback, get people to contribute, and make the entire group own it. Take an API-first approach to enable better collaboration.
Mature internal tools are potential candidates for open-source. Contribute back to the FOSS community with high-quality libraries/tools, and establish our brand.

Without the right amount of investment in sharpening our tools, it is harder for engineers to work together, potentially leading to the group’s failure. Group productivity, energy, and curiosity take precedence over business products. Well-designed engineering tools endorse group collaboration.