Using KBFS as a makeshift maven server

As I continue to write more and more Java libraries for personal and public use, I keep finding myself limited by my library hosting solutions. Maven servers are currently my go-to way of storing and organizing all things Java. I have gone through a solid handful of servers over the past few years, here are my comments on each:

• GitHub Releases
  • No dependabot integration
  • No easy way to get Gradle to load files directly from GitHub
• JitPack
  • Slow builds
  • No easy way to publish custom artifacts or use custom groups
  • Sometimes unusably long cache policy
• Ultralight
  • Has a file transfer limit
  • Uses my personal API keys to interact with GitHub
  • No way to automate package updates
• GitHub Packages
  • Requires users to authenticate even for public assets
  • Has a file transfer limit
  • Uses a separate maven url per project

As a student, I prefer not to do the sensible solution–spin up an Artifactory server–as that costs money I could be spending on coffee.

What makes a maven server special?

Really, not much. As outlined in my previous maven-related post, a maven server is just a simple webserver with a specific directory structure, and some metadata files placed in specific locations.

Let’s say we wanted to publish a package with the following attributes:

The resulting directory structure would end up looking like:

.
└── ca
    └── retrylife
        └── example
            └── example-artifact
                ├── maven-metadata.xml
                ├── maven-metadata.xml.sha1
                └── 1.0.4
                    ├── example-artifact-1.0.4.jar
                    ├── example-artifact-1.0.4.jar.sha1
                    ├── example-artifact-1.0.4.pom
                    └── example-artifact-1.0.4.pom.sha1

In this example. I chose to use the sha1 hashing algorithm, but maven clients support pretty much any algorithm I can think of.

As you can see, the files are layed out very logically. Packages are organized similarly to how you organize your source code; each artifact is accompanied by a Project Object Model describing it, maven-metadata files keep track of versioning, and every file also has a hash alongside it.

For reference, the maven-metadata.xml in this example would look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>ca.retrylife.example</groupId>
  <artifactId>example-artifact</artifactId>
  <versioning>
    <release>1.0.4</release>
    <latest>1.0.4</latest>
    <versions>
      <version>1.0.4</version>
    </versions>
    <lastUpdated>20210216203206</lastUpdated>
  </versioning>
</metadata>

As far as I know, maven-metadata is not actually required, but I always include them so that I can make use of dynamic versions in Gradle.

Using a static CDN as a maven server

Since there is nothing special about a maven server aside from its directory structure, anywhere that can host files can become a server. My choice for now is Keybase’s KBFS. KBFS is a pgp-signed file store that allows every user 250GB of free storage. This web filesystem is mounted to the user’s device using FUSE in a similar way to rclone.

This local mount & sync setup allows me to interact with my /keybase mountpoint like any other directory, while having all its contents automatically backed up and published.

Taking advantage of this

Gradle’s maven-publish plugin is designed to publish packages to remote servers, but will also work with local URIs. Simply pointing a MavenPublication to /keybase/public/ewpratten/maven/release (my directory of choice for now) will automatically generate everything mentioned in the section about file structure above.

My exact configuration for doing this in gradle is as follows (source):

apply plugin: "maven-publish"

// Determine SNAPSHOT vs release
def isRelease = !project.findProperty("version").contains("-SNAPSHOT")
if (!isRelease) {
    println "Detected SNAPSHOT"
}

publishing {
    repositories {
        maven {
            name = "KBFS"
            if (isRelease) {
                url = uri(
                    project.findProperty("kbfs.maven.release") ?: "/keybase/public/ewpratten/maven/release"
                )
            } else {
                url = uri(
                    project.findProperty("kbfs.maven.snapshot") ?: "/keybase/public/ewpratten/maven/snapshot"
                )
            }
        }
    }
}

This configuration is a bit fancy as it will separate snapshots from releases, and allow me to completely override the endpoint(s) in my settings.gradle file if I choose. A minimal approach would be:

apply plugin: "maven-publish"

publishing {
    repositories {
        maven {
            name = "KBFS"
            url = uri("/keybase/public/<your username>/maven")
        }
    }
}

Pretty URLs

With the solution outlined in this post, the end user would end up specifying one of the following URLs in their maven client:

• https://<username>.keybase.pub/maven/release/
• https://<username>.keybase.pub/maven/snapshot/

While that is perfectly fine, I prefer to keep all of my projects / services / etc under my personal domain (retrylife.ca). Unlike the rest of this post, this step does cost some money.

I already rent two servers for various other projects, and one of them is running the Caddy webserver and acting as a reverse proxy. I have pointed two domains (release.maven.retrylife.ca and snapshot.maven.retrylife.ca) at this server and am using the following rules to route them:

release.maven.retrylife.ca {
    route /* {
        rewrite * /maven/release/{path}
        reverse_proxy https://ewpratten.keybase.pub {
            header_up Host ewpratten.keybase.pub
        }
    }
}

snapshot.maven.retrylife.ca {
    route /* {
        rewrite * /maven/snapshot/{path}
        reverse_proxy https://ewpratten.keybase.pub {
            header_up Host ewpratten.keybase.pub
        }
    }
}

This means that I can point users at one of the following domains, and they will get the packages they are looking for:

• https://release.maven.retrylife.ca/
• https://snapshot.maven.retrylife.ca/

I am also now able to switch out backend servers / services whenever I want, and users will see no difference.

Future improvements

Some time in the future, I plan to move from KBFS to the S3-based DigitalOcean Spaces so I can speed up the download time for packages, and have better global distribution of files.