Troubleshooting Active Recording

Active Recording comes with some idiosyncracies all its own and will need specific things tested in order to troubleshoot. 

NOTE:  It is recommended that Avaya Call Reporting and ACR's Recording Library are on the voice network and that is the ideal configuration.  Recording Library is essentially a SIP handset, and should be on the same voice network as all the agent phones, SCN, etc.

Answer

Follow these steps in order to fix Active Recording while also gathering the most data necessary for Xima Support to look at if assistance is required.

Initial Checks

  1. Double-check that you are using OEM Avaya Call Reporting (ACR) with ACR Recording Library (RL) installed
    • Chronicall will not work with Active Recording, and all licensing needs to be correct
  2. Double-check your version of IP Office, older versions do not support Active Recording
    • Also, if you are not on IP Office 11.0.4.2 build 58 or higher, please update to that build, or past it
      • This update has fixes for a DEVLINK3_CONNECT_FAILED message that sometimes happens on the older versions
      • See the DEVLINK3_CONNECT_FAILED section at the bottom of this article for more info on those messages
    • You can check the version of the IP Office by logging into the IP Office Manager application and finding it two ways:
      1. When initially connecting to the phone system, the "Select IP Office" window will show the name and version number of all systems found on the network
      2. If already logged into the system via Manager, the version number of the system can be found by clicking on the Control Unit section on the left side, then examining the Version field on the right.
  3. Ensure there is no characters in the Trunk VRTX box, and that Auto Detect VRTX is disabled in Chronicall's Settings, and then restart the Recording Library service
  4. Update ACR to the latest available build
  5. If this is Multimedia/Contact Center, please make sure that the Transfer action in the VM Pro is not an Assisted Transfer

Troubleshooting Steps

ACR Configuration

  1. Follow the Understanding Active Recordings article to make sure that everything is configured correctly
    • NOTE: The Recording Library System ID must always be set to 1 for Active Recording to work
  2. Set the recording rules to a Basic Call Event with "Only Talking Events" set to "True"
    • This will help eliminate the Recording Rules as the point of failure
    • If possible, restart the ACR Server service and the RL service after making recording rule changes
  3. Within ACR, navigate to Admin (System) > System Settings > Advanced Settings > Log Filters > Recording Library
    • Turn on these Log Filters:
      • 'Recording Live Association'
      • 'Recording Ports'
      • 'DevLink Active Recording'
      • 'DevLink Active Recording Detail'
      • 'Recording Rules'

IPO Configuration

  1. Check in the IP Office Manager that the SIP Registrar is enabled
      • NOTE: This must be enabled on all nodes that have users that need to be recorded. 
    2. Log into the IPO's configuration within the IP Office Manager
    3. Go to System > LAN1 > VoIP
      • NOTE: This must be configured on LAN1 as LAN2 will fail to send RTP data
    4. Check the box for SIP Registrar Enable

      SIP_Registrar.png

    5. Save and Merge the changes
      • This will require a PBX reboot which should normally occur after-hours
  2. Restart services to ensure good IPO and ACR connection

Check Recording Library (RL) Server Ports

  1. Check if the ports being used by the RL are open
    1. Open a command line and use the following commands on the RL server
      • Windows
        • netstat -ano | Find "5060"
        • netstat -ano | Find "6970"
        • Example

          netstat.png

          • NOTE: Seeing Listening or *:* does not mean that the ports are completely open across their system or network, just that the RL server is listening for data on those ports
          • NOTE: Only the TCP will show "listen" or "established", UDP is the "connectionless" protocol
          • Running a telnet command to test the connection will work for 5060, but not 6970 as 6970 is UDP
      • Linux 
        • netstat -an | grep "5060"
        • netstat -an | grep "6970"
        • Example

          Linux_netstat.png

          • NOTE: Seeing Listening or *:* does not mean that the ports are completely open across their system or network, just that the RL server is listening for data on those ports
          • NOTE: Only the TCP will show "listen" or "established", UDP is the "connectionless" protocol
          • Running a telnet command to test the connection will work for 5060, but not 6970 as 6970 is UDP
    2. If they are not open and listening, your SysAdmin will need to open them

Check the Firewall

  1. Verify that the firewall, either server-based or network-based, is not blocking those ports 
    • NOTE: The two default ports to check on the Recording Library server are:
      • UDP Port 6970
      • TCP Port 5060
  2. You can help check which ports are open and what rules are being used by using the following commands on the Recording Library server 
      • Firewalld (Linux) 
        • systemctl status firewalld
          •  This command will check if firewalld is running on the server
        • sudo firewall-cmd --list-ports
          • This will list any ports that are currently whitelisted
      • IPtables (Linux) 
        • iptables -L
          • If iptables isn't running when you run the iptables -L command, you'll see what looks like empty tables.
      • Windows
        • There are so many different firewalls that they could be using, your best bet is to ask the other tech to doublecheck for you.
    • NOTE: These commands are meant to only steer in the right direction. Troubleshooting or changing a firewall is not a part of what we support here at Xima. Any changes should be made by the customer or a qualified person on their team.

Make Test Calls

  1. NOTE: If you have trouble following these steps or have any questions/concerns about them, it is recommended that you reach out to Xima Support.
  2. Set the Recording Library Logs to DEBUG
    1. Navigate to Admin (System) > System Settings > Recording Libraries  and select the ellipsis
    2. Select and "Edit" the "DevLink Recording - Recording Library" that you previously created
    3. Change "Service Logging Level" to DEBUG
  3. Turn on the Active Recording UCAPs
    1. Go to http://localhost:9080/diag?page=recordingPage from the main ACR server
      • You may need to log in using your ACR Administrator credentials
    2. Click the "System 1 On" button under the "Turn On Recording Raw Data"

      UCAPs_Turn_on.png

      • If this produces an error it's usually because it changes the IP address to 127.0.0.1, switch it back to localhost and it should work
  4. Make a test call to an agent that is set to be recorded
  5. Turn off Active Recording UCAPs when you are done testing
    1. Follow the previous steps but do "System 1 Off" instead
  6. Use the Log Aggregator Tool (left-hand side of the Diag page menu) to gather all logs
    1. Make sure 'Include UCAPs' is selected
    2. Hit 'Aggregate Logs'. This will automatically send Xima the logs
  7. Navigate to the Query BlueDB menu on the /diag site using Chrome
    1. Run a BlueDB query for the time you were testing
    2. Select "Download All" to create a .json file with all of the information for your test calls and send it to Xima Support

      BlueDB_Query.png

  8. Collect a screenshot of the test calls you made to provide Xima Support
    1. Navigate to Admin (System) > System Settings > Advanced Settings
      1. Ensure "Always Show Unassociated Recordings" and "Show Pending Recordings" are set to "True"
      2. Hit Save
    2. Make sure to remove any columns that won't help in order to clean up your screenshot
      • I.e. "Location" and "Caller Name" are generally useless for these tests

Next Steps if unresolved

Xima Support Agents: Please follow this article as well for gathering additional information, Further Active Recording Testing

In version 4.2(9am) the way in which we request recordings from the IP Office has changed. If you are past that build and still having issues please collect additional information for the Devs and Avaya to be able to troubleshoot further.

  • Please follow this article in order to gather the necessary System Monitor traces

DEVLINK3_CONNECT_FAILED

Since IP Office build 11.0.4.2 build 58, any time that a customer has experienced a DEVLINK3_CONNECT_FAILED message it has been either firewall or VCM resource-related

Ensure that H.323 Gatekeeper is checked

  • If this is not checked, it can also cause this error
  • This would be set on the PBX side, inside of Avaya Manager under LAN1 > VoIP > H.323 Gatekeeper Enable
    • Active Recording requires VCM Resources as VCM resources broker non-IP communications (TDM, Analog), etc to IP communications as well as IP to IP communications
    • You need VCM resources and the H.323 Gateway Enable selected
      • In Server Edition you don't have to account for VCM because they are already present, and note with IP500 you have to add the card and licenses

        mceclip0.png

Check for firewall-related issues

  • Please verify:
    1. That nothing is blocking the traffic between the Recording Library and IP Office
    2. Verify you are on a good version of IP Office (on or past IPO 11.0.4.2 build 58)
    3. Turn on DEBUG mode for the Recording Library along with the Log Filters recommended in the Make Test Calls section

Check for resource-related issues

  1. Open up the tomcat log and search for DEVLINK3_CONNECT_FAILED
    • These can be found within the tomcat/logs folder and usually end with either start.log or roll.log
    • Look for the most recent of those two types

      tomcat_log.png
  2. Around that error message should be IpAddress, which is the IP Office that we are making the request to

    Request_IP_Address.png

  3. If most calls record, but some fail to during peak times, this is most likely related to VCM resources and the partner will need to monitor their availability
    • NOTE: VM Pro and Auto Attendant take VCM resources, and each Active Recording also take a VCM
    • The partner can open the Avaya IP Office System Status application and look under the Alarms section to see if they have had any VCM or resource errors during their peak hours
    • Also bear in mind the total Active Recording port capacity of each IP Office

      Total_Active_Recording_Port_Capacity.png
  4.  If there is a need for more information from the IPO connection itself, refer to this article for more information on setting up System Monitor's trace filters.
Was this article helpful?
0 out of 1 found this helpful

Table of Contents

    Environment

    Operating System

    PBX System

    Chronicall Version