Troubleshooting Okta Advanced Server Access (ASA)

This post looks at the tools to use when troubleshooting issues with Okta Advanced Server Access (ASA). It’s not a “if you see this error, go do this” article – Google is great for that! This will look at where to go look for diagnostic info to help troubleshoot issues.

Article contents:

Revisiting the Okta Components and Flows

A typical ASA flow will involve the Okta Identity Cloud (for authentication), the ASA Cloud Service (for authorization and PKI), the ASA Client, an ASA Server Agent and maybe one or more Bastions/Gateways.

Main ASA Components

This article assumes you’re familiar with the components and their use. If not, have a look at the PAM (Incl. ASA) page.

There are a few standard flows for any SSH or RDP session in ASA (not including the common authN/authZ with Okta and ASA):

  • Client -> Target
  • Client -> Bastion -> Target
  • Client -> Gateway -> Target
  • Client -> (combinations of bastion/gateway) -> Target
This article does not specifically address the new AD-Joined feature that is in limited Early Access, but the Client -> Gateway -> Target flow will cover some of the components.

It is important to understand what flow you’re looking at. There are two places to determine the flow:

  • The ASA Audit report (Logging -> Audits in the ASA admin console) will tell you whether a connection has gone directly to a target or whether it’s been routed via one or more bastions.
  • The ASA Project configuration will tell you what Gateway selectors are used and the Gateways menu item will show the Hostname of the related Gateway.

The following screen shots show finding the gateway used by a project.

Gateway Selector in Project Configuration
Gateway by Label

Understanding the flows will tell you where you need to go (which components) to determine where something is breaking down. The following sections will look at the individual components.


Logs in Okta and the ASA Cloud Service

Given that the Okta Identity Cloud and ASA Cloud Service are SaaS services, we can only access the logging via the various admin consoles (or APIs). Okta will give you events related to user authentication. An example is shown below, showing sign-on to Okta, MFA prompts and SSO to ASA.

Events for User Authentication in Okta System Log

As with all Okta system log events you can expand each event and drill down into events and there’s a wealth of detail that may help troubleshooting (like timestamps and userids).

Drill Down to More Detail

The ASA audit trail in the ASA admin console will show ASA-specific events. In the example below you can see establishing a client session, issuing of certificates (for the two hops, client -> bastion, bastion -> target), and the hops (bastion and target).

ASA Audit Logging

Again, these event logs indicate IPs/hostnames (ASA hostnames), timestamps and userids.


Troubleshooting Issues with the Client

The client (sft) will be running on users workstations, connecting to the ASA cloud service, and calling the local SSH/RDP clients.

On a Mac or Linux system you should see logs in <user>/Library/Logs/ScaleFT/sft or equivalent. On a Windows system you should see the logs in <user>\AppData\Local\ScaleFT\logs.

You can also run the sft client in debug mode, as follows.

Windows Workstations

In Powershell
PS> C:\Users\<user>> $env:SFT_DEBUG="1"
PS> C:\Users\<user>> sft rdp <server_name>

# Windows CMD
C:\Users\<user>> set SFT_DEBUG="1"
C:\Users\<user>> sft rdp <server_name>

Linux Workstations

~ % SFT_DEBUG=1 sft rdp <server_name>

With SSH commands you can also pass the debug argument (-v = debug level 1, -vv = debug level 2 and -vvv = debug level 3). For example:

<user>@<machine> sft % ssh -v ubuntu-target
OpenSSH_8.6p1, LibreSSL 3.3.5
debug1: Reading configuration data /Users/<user>/.ssh/config
debug1: Executing command: '/usr/local/bin/sft resolve -q  ubuntu-target'
debug1: /Users/<user>/.ssh/config line 5: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Executing proxy command: exec "/usr/local/bin/sft" proxycommand  ubuntu-target
debug1: identity file /Users/<user>/.ssh/id_rsa type 0
...
debug1: Local version string SSH-2.0-OpenSSH_8.6
debug1: Remote protocol version 2.0, remote software version SFT-PROXY2
debug1: compat_banner: no match: SFT-PROXY2
debug1: Authenticating to ubuntu-target:22 as '<user>'
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:<blah>
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ubuntu-target' is known and matches the RSA host key.
debug1: Found key in /Users/<user>/Library/Application Support/ScaleFT/proxycommand_known_hosts:25
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/<user>/.ssh/id_rsa RSA ...
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentication succeeded (none).
Authenticated to ubuntu-target (via proxy).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: proc
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_AU.UTF-8"
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-1128-aws x86_64)

The above shows how the ssh alias is actually invoking sft, and sft running through its hostname resolution, proxy command and calling ssh with a cert.


Troubleshooting Issues with the Server Agent

The ASA server Agent shows up on systems as sftd (ScaleFT daemon).

Windows Servers

On a Windows server the logs files can be found in: C:\Windows\System32\config\systemprofile\AppData\Local\scaleft\Logs\sftd.

It is also worth reading https://help.okta.com/asa/en-us/Content/Topics/Adv_Server_Access/docs/windows.htm to understand the SSH tunnelling mechanism used on Windows.

Linux Servers

Logging with the ASA Linux server Agent can be different depending on the Linux platform. From a colleague “When the ASA agent is installed on a Linux server, it identifies if the server is running systemd (RHEL7+, etc.), and specifically the journald component of systemd. If it finds journald, the ASA agent will use systemd-journald.service for logging. If systemd is NOT present, ASA will fall back to whichever syslog server is available, i.e. syslog-ng or rsyslog“. For most Linux servers you can use the journalctl -u sftd command to see the logs. Otherwise, look in the /var/log/sftd/ folder.

You may also find useful information in system/security logs. For example, the following is from the /var/log/auth.log on an Ubuntu system and shows user activity via ssh.

Apr  1 01:11:37 localhost sshd[2153]: Accepted publickey for kent_brockman from 172.31.34.158 port 47648 ssh2: RSA-CERT ID rZUPREEytTqER4VPJOvzh5ewB0g= (serial 0) CA RSA SHA256:WzysIL+zhessHyJ8ZbeimAjL1NHE4bnSweblUU8+k6Q
Apr  1 01:11:37 localhost sshd[2153]: pam_unix(sshd:session): session opened for user kent_brockman by (uid=0)
Apr  1 01:11:37 localhost systemd: pam_unix(systemd-user:session): session opened for user kent_brockman by (uid=0)
Apr  1 01:11:37 localhost systemd-logind[1175]: New session 1 of user kent_brockman.
Apr  1 01:12:08 localhost sudo: kent_brockman : TTY=pts/0 ; PWD=/usr/common-scripts ; USER=root ; COMMAND=/usr/sbin/useradd jsmiff
Apr  1 01:12:08 localhost sudo: pam_unix(sudo:session): session opened for user root by kent_brockman(uid=0)
Apr  1 01:12:15 localhost sudo: kent_brockman : TTY=pts/0 ; PWD=/usr/common-scripts ; USER=root ; COMMAND=/usr/sbin/deluser jsmiff
Apr  1 01:12:15 localhost sudo: pam_unix(sudo:session): session opened for user root by kent_brockman(uid=0)
Apr  1 01:12:19 localhost sshd[2153]: pam_unix(sshd:session): session closed for user kent_brockman

If you plumb your system logs to a SIEM, it may be easier to search through events in the SIEM.


Troubleshooting Issues with the Gateway

Whilst an ASA Bastion is just another server running the sftd, an ASA Gateway is running a different service entirely – sft-gatewayd (although a gateway may also be running the agent, so you could see the sftd process running).

root@ip-172-31-35-163:/home/okta_admin# ps -ef | grep sft
root       804     1  0 00:59 ?        00:00:00 /usr/sbin/sftd
root       828     1  0 00:59 ?        00:00:00 /usr/sbin/sft-gatewayd service
sft-gat+   943   828  0 00:59 ?        00:00:00 /usr/sbin/sft-gatewayd proxy --log-level info
sftd       966   804  0 00:59 ?        00:00:00 /usr/sbin/sftd _broker

The gateways only run on Linux (see https://help.okta.com/asa/en-us/Content/Topics/Adv_Server_Access/docs/supported-os.htm).

The /etc/sft/sft-gatewayd.yaml file contains the gateway configuration settings, including log level and session capture storage directory (by default they are stored in /var/log/sft/sessions.).

root@ip-172-31-35-163:/etc/sft# cat sft-gatewayd.yaml 
# Setup token from ASA. This is required for the gateway to start correctly.
SetupToken: sft-gw.<remove-from-capture>

# Verbosity of the logs. info is the default and recommended. debug or error
# levels are also available.
LogLevel: info

# The network address clients will be instructed to use to access this gateway.
# AccessAddress: "1.1.1.1"
# The network port clients will be instructed to use to access this gateway.
# AccessPort: 7234

# The network address that the gateway will listen on.
# ListenAddress: "0.0.0.0"
# The network port that the gateway will listen on.
# ListenPort: 7234

# The directory where finalized session logs will be stored.
# SessionLogDir: "/var/log/sft/sessions"

# SessionLogFlushInterval controls how frequently logs for an active session
...

As with the Server Agent, there are no logs in the /var/log/sftd folder. You need to run journalctl -u sft-gatewayd to see the gateway logs.

The gateway logs can show a lot of information about traffic traversing the gateway.


This concludes this post on troubleshooting.

Leave a Reply