Journey of Integrating Gitlab through Automated Hooks

The Core Hurdle: Manual Setup and Customer Dependencies

Our aim was to reward team members for their code review participation through Hexmos Feedback. We reward on the basis of contributions done for a merge request in Gitlab. In our journey to integrate GitLab with our product, we encountered manual configuration, customer effort, and other obstacles.

We needed Feedback product’s customer action setting up webhooks for projects to receive merge request events and reward participants. Customers would have to set up a webhook like this for each project.

The main problem was linking the Gitlab merge request participants with the Hexmos users. It was difficult because the email addresses might not match. This made it hard to identify the participants. To solve the user linking problem, we added OAuth to the customer’s GitLab instance. Now, participants only need to provide OAuth once to receive rewards. The downside was that our customers had to set up the Oauth process by hand. This issue pushed us into a cycle of manual operations, increasing customer dependency.

We understood how manual steps could slow down our process and hard to scale up. It was clear that we needed a new solution. We aimed to move from a manual setup to a more automated and user-friendly approach. Below, we break down how we tackled each issue in different categories.

Reducing the manual tasks by sequencing APIs

To reduce the manual tasks and improve efficiency, we focused on sequencing the Gitlab APIs.

Webhook Setup Across Projects for Contributors Info

The Customer had to set up a webhook for each project so that we could get contributor info in Merge Requests. Customers had to take multiple actions, which made it difficult to grow and less efficient. A manual setup for each project would look like this.

We considered using Group webhooks to establish webhooks for all projects in a group. But, this feature came with a need for a paid license. This API would not work on customers who wouldn't have the Premium Gitlab. So we worked around it in a way such that it would work even for basic Gitlab. We want to give our customers affordable solutions, so we chose not to take this path. Instead, we opted for the Project Hook API to establish a webhook in each project. This helped us keep our solution affordable for customers.

Example of creating a webhook for a project:

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \
 --data "url=http://example.com/webhook&push_events=true" \
 "https://selfhosted.gitlab.example.com/api/v4/projects/:id/hooks"

This resolved the manual webhook configuration issue. Yet, the webhook hid the participants’ email and even though the email was visible we wouldn’t know whom to reward, resulting in a new problem.

Mapping Gitlab Users to Hexmos Users

The webhook didn't show participants' email addresses, only their GitLab participants' IDs. We needed the participants' email so that we could reward them in the MR process.

Example of a participant who commented on an MR:

{
"id": 15281990,
"name": "Jeff Bezz",
"email": "[REDACTED]"
}

To get their email it required participants' Authorization. We would not know whom to reward as the registered email might differ in Gitlab and Hexmos. So, we set up OAuth integration for our web app. This was crucial for mapping GitLab users with Hexmos.

Eradicating Manual Application Configuration for OAuth Integration

To solve the problem of not having enough data, we used OAuth integration to get participants’ information. Yet, this required the app to be configured within the GitLab instance. To set up the app, the admin user needed to configure it. This was the manual step that customers had to go through.

To skip this step, we asked an admin user with the right access for a Personal Access Token (PAT). PAT allows actions on the user's behalf without needing their manual action. We used PAT to create an Application for our web app in their GitLab.

Example to create an app using Application API:

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \
 --data "name=MyApplication&redirect_uri=http://redirect.uri&scopes=api read_user email" \
 "https://selfhosted.gitlab.example.com/api/v4/applications"

With the PAT, we configured the Application. Now participants of the same Gitlab instance could simply provide their Gitlab ID and email with a button click.

The above action would redirect to their Gitlab and ask for Authorization to share the user data.

When the user gives permission, Hexmos will connect the Gitlab user data with Hexmos user data. This connection allows us to reward the Gitlab user for participating in the Merge request. The webhook setup still needed multiple admin user actions. In the next section, we tackle this challenge.

Removing Dependency on Users for Webhook Configurations

Customers were required to create a webhook for each project, which was burdensome. So we fetched all groups the admin user is in using the List Groups API. First, list all the projects in the group or subgroups. Then, created the webhook for them.

Example to list groups

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://selfhosted.gitlab.example.com/api/v4/groups"

Example to list sub-groups

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://selfhosted.gitlab.example.com/api/v4/groups/:id/subgroups"

Next, we list all projects in the group using the Group’s Projects API

Example to list projects of the group:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://selfhosted.gitlab.example.com/api/v4/groups/:id/projects"

Finally, we implemented the webhook for each project using the Project’s Hook API

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \
 --data "url=http://example.com/webhook&push_events=true" \
 "https://selfhosted.gitlab.example.com/api/v4/projects/:id/hooks"

Using automated API calls, we found all the groups and projects linked to the admin user. Then, we set up webhooks for each project without needing an admin user to do it.

Transition to Automated Efficacy: Reaping the Rewards of API Sequencing

By using API sequencing, we were able to address manual tasks and reduce the burden on customers. This improved our GitLab integration by automating and simplifying operations. This post demonstrates how our solutions enhance performance, user experience, and capacity.