This post looks at the tools to use when troubleshooting issues with Okta Advanced Server Access (ASA). It’s not a “if you see this error, go do this” article – Google is great for that! This will look at where to go look for diagnostic info to help troubleshoot issues.
Revisiting the Okta Components and Flows
A typical ASA flow will involve the Okta Identity Cloud (for authentication), the ASA Cloud Service (for authorization and PKI), the ASA Client, an ASA Server Agent and maybe one or more Bastions/Gateways.
This article assumes you’re familiar with the components and their use. If not, have a look at the PAM (Incl. ASA) page.
There are a few standard flows for any SSH or RDP session in ASA (not including the common authN/authZ with Okta and ASA):
- Client -> Target
- Client -> Bastion -> Target
- Client -> Gateway -> Target
- Client -> (combinations of bastion/gateway) -> Target
This article does not specifically address the new AD-Joined feature that is in limited Early Access, but the Client -> Gateway -> Target flow will cover some of the components.
It is important to understand what flow you’re looking at. There are two places to determine the flow:
- The ASA Audit report (Logging -> Audits in the ASA admin console) will tell you whether a connection has gone directly to a target or whether it’s been routed via one or more bastions.
- The ASA Project configuration will tell you what Gateway selectors are used and the Gateways menu item will show the Hostname of the related Gateway.
The following screen shots show finding the gateway used by a project.
Understanding the flows will tell you where you need to go (which components) to determine where something is breaking down. The following sections will look at the individual components.
Logs in Okta and the ASA Cloud Service
Given that the Okta Identity Cloud and ASA Cloud Service are SaaS services, we can only access the logging via the various admin consoles (or APIs). Okta will give you events related to user authentication. An example is shown below, showing sign-on to Okta, MFA prompts and SSO to ASA.
As with all Okta system log events you can expand each event and drill down into events and there’s a wealth of detail that may help troubleshooting (like timestamps and userids).
The ASA audit trail in the ASA admin console will show ASA-specific events. In the example below you can see establishing a client session, issuing of certificates (for the two hops, client -> bastion, bastion -> target), and the hops (bastion and target).
Again, these event logs indicate IPs/hostnames (ASA hostnames), timestamps and userids.
Troubleshooting Issues with the Client
The client (
sft) will be running on users workstations, connecting to the ASA cloud service, and calling the local SSH/RDP clients.
On a Mac or Linux system you should see logs in
<user>/Library/Logs/ScaleFT/sft or equivalent. On a Windows system you should see the logs in
You can also run the sft client in debug mode, as follows.
In Powershell PS> C:\Users\<user>> $env:SFT_DEBUG="1" PS> C:\Users\<user>> sft rdp <server_name> # Windows CMD C:\Users\<user>> set SFT_DEBUG="1" C:\Users\<user>> sft rdp <server_name>
~ % SFT_DEBUG=1 sft rdp <server_name>
With SSH commands you can also pass the debug argument (
-v = debug level 1,
-vv = debug level 2 and
-vvv = debug level 3). For example:
<user>@<machine> sft % ssh -v ubuntu-target OpenSSH_8.6p1, LibreSSL 3.3.5 debug1: Reading configuration data /Users/<user>/.ssh/config debug1: Executing command: '/usr/local/bin/sft resolve -q ubuntu-target' debug1: /Users/<user>/.ssh/config line 5: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files debug1: /etc/ssh/ssh_config line 54: Applying options for * debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling debug1: Executing proxy command: exec "/usr/local/bin/sft" proxycommand ubuntu-target debug1: identity file /Users/<user>/.ssh/id_rsa type 0 ... debug1: Local version string SSH-2.0-OpenSSH_8.6 debug1: Remote protocol version 2.0, remote software version SFT-PROXY2 debug1: compat_banner: no match: SFT-PROXY2 debug1: Authenticating to ubuntu-target:22 as '<user>' debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: firstname.lastname@example.org debug1: kex: host key algorithm: rsa-sha2-512 debug1: kex: server->client cipher: email@example.com MAC: <implicit> compression: none debug1: kex: client->server cipher: firstname.lastname@example.org MAC: <implicit> compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: SSH2_MSG_KEX_ECDH_REPLY received debug1: Server host key: ssh-rsa SHA256:<blah> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory debug1: Host 'ubuntu-target' is known and matches the RSA host key. debug1: Found key in /Users/<user>/Library/Application Support/ScaleFT/proxycommand_known_hosts:25 debug1: rekey out after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey in after 134217728 blocks debug1: Will attempt key: /Users/<user>/.ssh/id_rsa RSA ... debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentication succeeded (none). Authenticated to ubuntu-target (via proxy). debug1: channel 0: new [client-session] debug1: Entering interactive session. debug1: pledge: proc debug1: Sending environment. debug1: channel 0: setting env LANG = "en_AU.UTF-8" debug1: client_input_global_request: rtype email@example.com want_reply 0 Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-1128-aws x86_64)
The above shows how the ssh alias is actually invoking sft, and sft running through its hostname resolution, proxy command and calling ssh with a cert.
Troubleshooting Issues with the Server Agent
The ASA server Agent shows up on systems as
sftd (ScaleFT daemon).
On a Windows server the logs files can be found in:
It is also worth reading https://help.okta.com/asa/en-us/Content/Topics/Adv_Server_Access/docs/windows.htm to understand the SSH tunnelling mechanism used on Windows.
Logging with the ASA Linux server Agent can be different depending on the Linux platform. From a colleague “When the ASA agent is installed on a Linux server, it identifies if the server is running systemd (RHEL7+, etc.), and specifically the journald component of systemd. If it finds journald, the ASA agent will use systemd-journald.service for logging. If systemd is NOT present, ASA will fall back to whichever syslog server is available, i.e. syslog-ng or rsyslog“. For most Linux servers you can use the
journalctl -u sftd command to see the logs. Otherwise, look in the
You may also find useful information in system/security logs. For example, the following is from the
/var/log/auth.log on an Ubuntu system and shows user activity via ssh.
Apr 1 01:11:37 localhost sshd: Accepted publickey for kent_brockman from 172.31.34.158 port 47648 ssh2: RSA-CERT ID rZUPREEytTqER4VPJOvzh5ewB0g= (serial 0) CA RSA SHA256:WzysIL+zhessHyJ8ZbeimAjL1NHE4bnSweblUU8+k6Q Apr 1 01:11:37 localhost sshd: pam_unix(sshd:session): session opened for user kent_brockman by (uid=0) Apr 1 01:11:37 localhost systemd: pam_unix(systemd-user:session): session opened for user kent_brockman by (uid=0) Apr 1 01:11:37 localhost systemd-logind: New session 1 of user kent_brockman. Apr 1 01:12:08 localhost sudo: kent_brockman : TTY=pts/0 ; PWD=/usr/common-scripts ; USER=root ; COMMAND=/usr/sbin/useradd jsmiff Apr 1 01:12:08 localhost sudo: pam_unix(sudo:session): session opened for user root by kent_brockman(uid=0) Apr 1 01:12:15 localhost sudo: kent_brockman : TTY=pts/0 ; PWD=/usr/common-scripts ; USER=root ; COMMAND=/usr/sbin/deluser jsmiff Apr 1 01:12:15 localhost sudo: pam_unix(sudo:session): session opened for user root by kent_brockman(uid=0) Apr 1 01:12:19 localhost sshd: pam_unix(sshd:session): session closed for user kent_brockman
If you plumb your system logs to a SIEM, it may be easier to search through events in the SIEM.
Troubleshooting Issues with the Gateway
Whilst an ASA Bastion is just another server running the
sftd, an ASA Gateway is running a different service entirely –
sft-gatewayd (although a gateway may also be running the agent, so you could see the sftd process running).
root@ip-172-31-35-163:/home/okta_admin# ps -ef | grep sft root 804 1 0 00:59 ? 00:00:00 /usr/sbin/sftd root 828 1 0 00:59 ? 00:00:00 /usr/sbin/sft-gatewayd service sft-gat+ 943 828 0 00:59 ? 00:00:00 /usr/sbin/sft-gatewayd proxy --log-level info sftd 966 804 0 00:59 ? 00:00:00 /usr/sbin/sftd _broker
The gateways only run on Linux (see https://help.okta.com/asa/en-us/Content/Topics/Adv_Server_Access/docs/supported-os.htm).
/etc/sft/sft-gatewayd.yaml file contains the gateway configuration settings, including log level and session capture storage directory (by default they are stored in
root@ip-172-31-35-163:/etc/sft# cat sft-gatewayd.yaml # Setup token from ASA. This is required for the gateway to start correctly. SetupToken: sft-gw.<remove-from-capture> # Verbosity of the logs. info is the default and recommended. debug or error # levels are also available. LogLevel: info # The network address clients will be instructed to use to access this gateway. # AccessAddress: "22.214.171.124" # The network port clients will be instructed to use to access this gateway. # AccessPort: 7234 # The network address that the gateway will listen on. # ListenAddress: "0.0.0.0" # The network port that the gateway will listen on. # ListenPort: 7234 # The directory where finalized session logs will be stored. # SessionLogDir: "/var/log/sft/sessions" # SessionLogFlushInterval controls how frequently logs for an active session ...
As with the Server Agent, there are no logs in the
/var/log/sftd folder. You need to run
journalctl -u sft-gatewayd to see the gateway logs.
The gateway logs can show a lot of information about traffic traversing the gateway.
This concludes this post on troubleshooting.