Automatically mirror Gitlab repositories to Github

Please contact us if anything is not clearly described, does not work, seems incorrect or if you require support.

Table of contents

• The Problem with mirroring from Gitlab to Github

• Installing the Gitlab CLI glab

• Installing the Github CLI gh

• Setting up the Gitlab to Github mirror script

The Problem with mirroring from Gitlab to Github

We want to mirror some of our public Gitlab repositories to github for better visibility. There is a gitlab function available for that at Settings -> Repository -> Repository mirroring, which is documented here, but sadly it gives this error when used with github:

13: push to mirror: git push: exit status 128, stderr: "remote Support for password authentication was removed on August 13, 2021.\nremote: Please see https://docs.github.com/en/getting-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.\nfatal: Authentication failed for: 'https://github.com/Blunix-GmbH/role-haproxy/'\n".

Installing the Gitlab CLI glab

So lets just script it. This will not be triggered automatically, but you can setup a cronjob on your gitlab server that does it once a day. Gitlab has a CLI tool for administration. A .deb package can be downloaded for Ubuntu and Debian Linux based workstations at the gitlab.com download page. For most Ubuntu and Debian based servers, choose "glab__Linux_x86_64.deb".

wget https://gitlab.com/gitlab-org/cli/-/releases/v1.36.0/downloads/glab_1.36.0_Linux_x86_64.deb
sudo dpkg -i glab_1.36.0_Linux_x86_64.deb

After installation you have to authenticate against your gitlab instance. This configuration will be saved below "~/.config/glab-cli/". In order to authenticate you will need a token, which can be generated here:

https://your-gitlab-domain.com/-/user_settings/personal_access_tokens

You only need to select the "api" and "read_repository" permission:

glab auth login
? What GitLab instance do you want to log into? GitLab Self-hosted Instance
? GitLab hostname: git.blunix.com
? API hostname: git.blunix.com
- Logging into git.blunix.com
? How would you like to login? Token

Tip: you can generate a Personal Access Token here https://git.blunix.com/-/profile/personal_access_tokens?scopes=api,write_repository
The minimum required scopes are 'api' and 'write_repository'.
? Paste your authentication token: **************************
? Choose default git protocol SSH
? Choose host API protocol HTTPS
- glab config set -h git.blunix.com git_protocol ssh
✓ Configured git protocol
- glab config set -h git.blunix.com api_protocol https
✓ Configured API protocol

If we check it now it wont work:

glab repo list
Showing 0 of 0 projects (Page 1 of 1)

For whatever reason the configuration created by "glab auth login", which is located at "~/.config/glab-cli/config.yml", is setup to only talk to gitlab.com by default, not your personal gitlab instance. Lets fix that by replacing all occurances of gitlab.com with your Gitlabs hostname:

# Change this to your gitlab instance
host: git.blunix.com
[...]

hosts:
    # Change this too
    git.blunix.com:
        [...]
        # And this as well
        api_host: git.blunix.com
        [...]

Or in one line:

sed -i 's/gitlab.com/git.blunix.com/g' ~/.config/glab-cli/config.yml

Now the repositories should be visible:

glab repo list
Showing 30 of 372 projects (Page 1 of 12)

ansible-roles/role-imgproxy                git@git.blunix.com:ansible-roles/role-imgproxy.git                Ansible role to install and configure imgproxy on Debian Linux                                                   
ansible-roles/role-golang                  git@git.blunix.com:ansible-roles/role-golang.git                  Ansible role to install and configure golang on Debian Linux    
ansible-roles/role-systemd-journal-remote  git@git.blunix.com:ansible-roles/role-systemd-journal-remote.git  Ansible role to install and configure systemd-journal-remote on Debian Linux                                     
[...]

Installing the Github CLI gh

Next we need a cli client for github to automatically create repositores there. There is a github cli called "gh", which can simply be installed by apt:

sudo apt install gh

The configuration is very similar to the glab cli tool (but works right away). In our case, gitlab is not able to start a browser from the cli, so we copy paste the auth code to the URL displayed:

gh auth login
? What account do you want to log into? GitHub.com
? What is your preferred protocol for Git operations? HTTPS
? Authenticate Git with your GitHub credentials? Yes
? How would you like to authenticate GitHub CLI? Login with a web browser

! First copy your one-time code: D946-B977
Press Enter to open github.com in your browser... 
2024/02/15 23:44:52.178314 cmd_run.go:1055: WARNING: cannot start document portal: Expected portal at "/run/user/1000/doc", got "/home/user/.cache/doc"
/user.slice/user-1000.slice/session-2.scope is not a snap cgroup
! Failed opening a web browser at https://github.com/login/device
  exit status 4
  Please try entering the URL in your browser manually

✓ Authentication complete.
- gh config set -h github.com git_protocol https
✓ Configured git protocol
✓ Logged in as blunix0815

Lets check if the setup was successful:

gh repo list

There are no repositories in @blunix0815

gh repo list Blunix-GmbH

Showing 30 of 47 repositories in @Blunix-GmbH

Blunix-GmbH/role-borgbackup-server            Ansible role to install and configure borgbackup server on Debian Linux 
[...]

Setting up the Gitlab to Github mirror script

The following script uses the "glab" and "gh" cli tools to iterate over all repositories on the Blunix gitlab instance, create the same project on github, clone the project from gitlab and push the master or main branch to github.

#!/bin/bash
#
# Replicate all repositories from https://git.blunix.com/ansible-roles/ to https://github.com/Blunix-GmbH/


# List all repos below https://git.blunix.com/ansible-roles/
# Example output of glab repo list:
# ansible-roles/role-ssh                          git@git.blunix.com:ansible-roles/role-ssh.git                          Ansible role for managing OpenSSH-Server and users authorized_keys
glab repo list --all --group ansible-roles --per-page 1000 | while read line; do

    # Skip "Showing of 51 projects (Page 1 of 1)"
    [[ "$line" == Showing* ]] && continue
    # Skip empty lines
    [[ "$line" == "" ]] && continue

    # Extract repo name
    repo_name=$(echo $line | cut -d ' ' -f 1 | cut -d '/' -f 2)

    # Extract repo url
    repo_url=$(echo $line | cut -d ' ' -f 2)

    # Extract description
    repo_description=$(echo $line | cut -d ' ' -f 3-)

    # Give the user some information about current repo
    echo "Processing repo \"$repo_name\" with url \"$repo_url\""

    # Abort on empty description
    if [[ "$repo_description" == "" ]]; then
        echo "Description empty, aborting!"
        exit 1
    fi

    # Create repo on github
    gh repo create --disable-issues --disable-wiki --homepage https://www.blunix.com --public --description "$repo_description" Blunix-GmbH/$repo_name

    # Clone repo from gitlab
    git clone $repo_url

    # Push to github
    cd $repo_name
    git remote add github git@github.com:Blunix-GmbH/$repo_name
    git push github master
    cd ..
    rm -rf $repo_name

done

Make it executable like so:

chmod 700 /usr/local/sbin/sync-repos-to-github.sh

The following command will setup a cronjob for the script that will be executed daily at 5am:

(crontab -l 2>/dev/null; echo "0 5 * * * /usr/local/sbin/sync-repos-to-github.sh") | crontab