Introduction

SSH agent forwarding is a very useful tool. It allows a developer to connect from a local machine to a remote server, and then from that server to another, like a Git repository. This is done without copying private keys to the remote server. It makes work secure and easy.

However, sometimes it does not work as expected. A developer might see the error Could not open a connection to your authentication agent on the remote server, even when the configuration seems correct. This article will look at a real case. We will examine the logs and find a hidden reason for this problem. We will also learn how to fix it.

The Setup: Everything Looks Correct

Let’s start by looking at the configuration files and commands. This helps us understand the situation.

First, the local SSH configuration in ~/.ssh/config has a special setup for the remote server, which is named gitlab-runner.

Host gitlab-runner
    Hostname i-072fd6721a00f4b42
    ProxyCommand aws ssm start-session ...
    User dave
    IdentityFile ~/.ssh/id_ec2
    ForwardAgent yes
    ...

The line ForwardAgent yes clearly tells the SSH client to enable agent forwarding. This is the correct setting.

Next, on the local machine, we check if the SSH agent is running and has a key.

daveking@192 personal % ssh-add -l
256 SHA256:bPvs3l3EcWZup7NFtZ/KiOCdTyBSKknl+xwXt4ybFG8 daveking@qq.io (ED25519)

This output confirms the local agent has an identity loaded. So far, the local setup seems perfect.

The Remote Server: Also Configured Correctly

The problem could be on the remote server. The server’s SSH daemon (sshd) must allow agent forwarding. We can check its configuration.

dave@gitlab-runner:~$ grep Allow /etc/ssh/sshd_config.d/60-cloudimg-settings.conf 
AllowAgentForwarding yes

The server’s configuration shows AllowAgentForwarding yes. This means the server is ready to accept agent forwarding requests. This rules out a server-side misconfiguration, which is a common cause of this issue.

At this point, both the local client and the remote server seem to be configured correctly. Yet, the problem remains.

The Real Reason: A Hidden Detail in the Logs

To find the real reason, we need to look closely at the verbose output from the SSH connection command (ssh -vvv). There is one line that gives us a big clue.

debug1: auto-mux: Trying existing master at '/Users/daveking/.ssh/sockets/dave@i-072fd6721a00f4b42:22'

This line talks about “auto-mux” and a “master” connection. This refers to SSH’s Connection Multiplexing feature. This is a performance feature. It allows multiple SSH sessions to the same server to reuse a single, existing network connection. This makes new sessions start much faster.

Here is the trap: the settings for all new sessions are taken from the first master connection. Imagine the first time a connection was made to gitlab-runner. Maybe at that time, ForwardAgent yes was not in the config file, or no keys were added to the local agent. That first master connection was created without agent forwarding.

Even after updating the config file and adding keys to the agent, any new ssh gitlab-runner command will find and reuse that old master connection. Because the old connection does not have agent forwarding enabled, the new session will also fail to forward the agent.

The Solution: Resetting the Master Connection

The solution is simple once we understand the problem. We need to close the old master connection and start a new one with the correct settings.

The process involves these steps:

1. Close the existing master connection. This can be done with a special SSH command.

   ssh -O exit gitlab-runner
   

   The -O exit command tells SSH to find the master connection for the host gitlab-runner and terminate it.

2. Confirm the local agent is ready. It is always a good practice to check that the key is still loaded.

   ssh-add -l
   

3. Create a new connection. Now, simply connect to the server again.

   ssh gitlab-runner
   

   This command will now create a brand new master connection. Because our current configuration has ForwardAgent yes and the agent has a key, this new connection will have agent forwarding enabled correctly.

4. Verify on the remote server. Once logged in, check the agent status again.

   dave@gitlab-runner:~$ ssh-add -l
   256 SHA256:bPvs3l3EcWZup7NFtZ/KiOCdTyBSKknl+xwXt4ybFG8 daveking@qq.io (ED25519)
   

   Success! The key from the local machine is now visible on the remote server.

Here is a diagram that shows the troubleshooting logic we followed.

Conclusion

SSH is a powerful tool with many features designed to make work easier and more secure. However, features like connection multiplexing can sometimes lead to confusing behavior. When troubleshooting issues like a failing SSH agent forward, it is important to look at all the pieces: the local client, the remote server, and the connection process itself.

The key lesson here is that when SSH settings are changed, existing master connections might not use them. Resetting the master connection is a simple but very effective way to solve these kinds of problems. By understanding these details, everyone can use tools like SSH more effectively.