Custom Root Certificates for Git and Azure DevOps Agents
Corporate PKI is one of those things that usually stays invisible until a tool refuses to talk to an internal server.
The typical setup is familiar enough: a company has its own root certificate authority, one or more intermediate CAs, and internal HTTPS sites whose certificates chain back to that private root. Browsers can be taught to trust that chain. Command-line tools, build agents and bundled runtimes are less easily impressed.
This is a refreshed version of a 2016 recipe. The original problem is still recognisable: Git and a build agent need to connect to an internal TFS or Azure DevOps Server instance over HTTPS, but the machine does not yet trust the private root certificate. The names and defaults have changed since then. VSTS became Azure DevOps, the old cross-platform agent has moved on, and Ubuntu 14.04 belongs in the historical cabinet with Internet Explorer 11 and other period furniture.
The principle has not changed: install the certificate into the trust store used by the tool that is failing. The awkward part is that “the trust store” is not always singular.
Export the Right Certificate
Start with the certificate chain for the internal HTTPS site. For a server such as https://my.site.com/tfs, the chain might look like this:
Company Root CA
Company Issuing CA
my.site.com
For trust configuration you usually want the root CA, and sometimes the intermediate CA as well. You do not normally want only the leaf certificate for my.site.com, because that certificate will be renewed.
Export the certificate as Base-64 encoded X.509. The file may have a .cer, .crt or .pem extension depending on the tool, but the content should look like PEM:
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
If the file is binary DER instead, convert or re-export it as PEM. The extension alone is not proof. Certificates, like build scripts, enjoy looking correct right up until the moment they are not.
Git on Windows
For Git on Windows, the cleanest modern option is often to use the Windows certificate store through Schannel:
git config --global http.sslBackend schannel
That lets Git trust certificates already installed in the Windows certificate store, including your organisation’s root CA if Group Policy or manual installation has put it there.
If you want to keep the trust decision scoped to one internal server instead, point Git at a certificate file for that URL:
git config --global http."https://my.site.com/".sslCAInfo C:/certificates/company-root-ca.pem
You can check the value afterwards:
git config --global --get http."https://my.site.com/".sslCAInfo
Avoid http.sslVerify false. It is tempting in exactly the way bad shortcuts usually are: fast, quiet and waiting to become someone else’s incident report.
Linux Trust Store
On Ubuntu and Debian-style systems, add a local CA certificate by copying a PEM-encoded certificate with a .crt extension into /usr/local/share/ca-certificates, then rebuild the trust bundle:
sudo cp company-root-ca.crt /usr/local/share/ca-certificates/company-root-ca.crt
sudo update-ca-certificates
The output should say that a certificate was added. You can then test the TLS handshake with:
curl -v https://my.site.com/tfs
An HTTP 401 or 403 can still be a success for this test. It means TLS worked and the server is now arguing about identity or permissions instead of certificate trust, which is at least a more honest disagreement.
Azure DevOps and TFS Agents
The 2016 article used the VSTS cross-platform agent on Ubuntu 14.04. For a current setup, use Microsoft’s current self-hosted Azure Pipelines agent documentation and pick an operating system supported by the agent version you need.
The first trust step is still the operating-system certificate store. Microsoft documents this directly for self-hosted agents with self-signed or private CA certificates: install the server certificate chain into the machine trust store, then verify with a tool such as curl on Linux.
After that, configure the agent with the current config.sh workflow rather than downloading an old hard-coded agent archive:
mkdir ~/azdo-agent
cd ~/azdo-agent
# Download the current agent package from your Azure DevOps Server or Azure DevOps Services UI.
# Then unpack it here and configure it.
./config.sh --url https://my.site.com/tfs --pool Default --agent linux-agent-01
./run.sh
For a long-running machine, install it as a service after configuration:
sudo ./svc.sh install
sudo ./svc.sh start
There is one more wrinkle. Some Azure DevOps tasks run on Node.js, and Node can use its own trusted CA list. Microsoft calls out NODE_EXTRA_CA_CERTS for cases where Node-based tasks still fail with errors such as SELF_SIGNED_CERT_IN_CHAIN or unable to get local issuer certificate.
Point it at a PEM file containing the root and any required intermediate certificates:
export NODE_EXTRA_CA_CERTS=/etc/ssl/certs/company-ca-bundle.pem
If the agent runs as a service, make the environment variable part of the service environment and restart the agent. Otherwise the interactive shell will understand the certificate while the service continues to live in its own little weather system.
What Aged
The old recipe had three very 2016 ingredients: Internet Explorer 11, Ubuntu 14.04 and a specific VSTS agent tarball. Those details are now historical. The useful part was the shape of the problem:
- find the root or issuing CA that signs the internal server certificate
- install it into the trust store used by the failing tool
- verify TLS separately from authentication
- avoid disabling certificate validation as a “fix”
Custom root certificates are not glamorous. Done well, they are plumbing. Done badly, they become folklore, and folklore is a poor dependency for a build pipeline.
RSS - Posts
Spot on with this write-up, I really assume this web site needs rather more consideration. I’ll probably be again to learn much more, thanks for that info.