VRTX Troubleshooting

Overview

This is Xima's article for in-depth troubleshooting for the Xima VRTX recording hardware.  A VRTX is a line-side device that captures all inbound and outbound calls across a single PRI or SIP trunk.  It does not have the ability to capture internal calls since it looking directly at the inbound and outbound trunks.

This article covers many of the common issues, and how to diagnose them. It is recommended that there is a maximum of 2 VRTX devices per server, and also that they are on a physical server with a dedicated USB port for each device.  A USB hub is not recommended.

Diagnosis

Verify Using Chronicall Status and Cradle to Grave

1. Make sure your users are set up to be recorded via the internal recording rules for VRTX
   • Use this link to verify your current ones, or create a new recording rule.
     • Note:  We recommend setting up a recording rule with a simple list of agents to be recorded, and to record only talking events.
2. Verify the overall status of Chronicall and the Recording Library
   1. On the Chronicall Home Screen, left-click the Bell in the top right corner as shown below
      
      
   2. The bell may indicate that there is an issue upfront with a specific VRTX device
      • If everything is working it should look similar to the screen below with no red errors
      • If not, you will see any VRTX's with an issue below the Recording Library status
        
        
      • NOTES:
        • If you have recurring issues, verify the VRTX devices are not running through an external USB Hub
        • Be wary of unplugging a USB hub on a virtual host, as this can cause issues with the USB mapping
        • We recommend 2 VRTX's per Recording Library Server with their own dedicated USB ports
        • We also highly recommend keeping the VRTX's on a physical server and not a virtual server if possible
        • Keep in mind that the VRTX is only compatible with 10/100 lines. If you are using gigabit lines, the VRTX would see 0 data and 0 packets
3. Next, to help identify what issue is affecting your audio recordings, go to Reporting > Cradle to Grave
   
   
   
   
4. Select the previous business day for your date, and hit Execute
   • This will display all the calls in Chronicall's database for yesterday
     
     
     
5. Cradle to Grave will help you discover if you are getting all recordings, some recordings, or no recordings at all
   1. It will also show if you are logging all calls, from all your PBX sites. 
      • If you are missing calls in Cradle to Grave please refer to our Not Logging article. 
      • If you see a grayed-out recording icon, that means the call met a recording rule but didn't capture if Show Pending Recordings is set to True.
      • You can find this setting if logged in as an Administrator to verify it is enabled in Admin (System) > System Settings > Advanced Settings.  
        
        
        
        
        • In the screen below, you can see a call with a blue and white play icon that was captured via the VRTX.  Below that is a gray icon, indicating the call met a recording rule but did not capture
          
          
   2. Keep in mind the VRTX does not capture internal calls.  It will only keep inbound and outbound recordings on the Trunk (physical phone line) it is attached too, based on your recording rules
      • You will not see any icon for internal calls
      • It can be helpful to replicate your recording rule in Cradle to Grave.  For example, you can set Call DirectionInbound and outbound calls only, and only with the agents that should be recorded, and by the event Talking 
6. Make sure you are Always Show Unassociated Recordings is set to True while troubleshooting.  Then search in Cradle to Grave for the previous day to see if you see any recordings that have a play icon but are not association with a call (cannot be expanded like other calls with a +)
   1. Click here to see how to display unassociated recordings. 
   2. It normally takes around 4 hours for unassociated recordings to show in Cradle to Grave per our default VRTX settings, so that is why we like to look at a previous day.
   3. NOTES:
      • Chronicall and Recording Library are two separate processes and services.
      • Chronicall will log the call data while the Recording Library records the call
      • Once the Recording Library has finished recording a call it will then use the recording's metadata to associate with a call record in the database
      • If it cannot find a call to associate with, and "Keep Unassociated" is set to True within the Recording configuration, then the recording will be listed separately as a "Call Recorded" in Cradle to Grave and not have a + sign next to its line in Cradle to Grave like other calls.  
7. If calls are coming in as unassociated you will usually see the recording listed right above or below the call with the grayed-out play icon
   • The most common reason for this is due to the Trunk or the Channel being configured incorrectly (see the VRTX Configuration section) or a time difference between the audio recording and the time for the call in the database
     • The latter is usually caused if there is a time difference between the Chronicall, Recording Library, and PBX servers of more than a few seconds.  Ensure all 3 servers are on the exact same time stamp.
   • Note For Avaya CM customers
     • If you are using Avaya CM (Communication Manager) and notice unassociated recordings, please refer to this article as an additional step to this article's troubleshooting flow
     • Also with Avaya CM, some of those unassociated calls could be in the Call Detail View
8. Are there multiple VRTX's and Trunks (phone lines) being captured?  If so it is helpful to determine if a specific VRTX is having issues. 
   1. You can right-click on the column title bar in Cradle to Grave, and select Trunk from the Drop Down Box to show trunk info in Cradle to Grave
   2. It helps to drag the Recording column next to the Trunk column to diagnose which trunks may be having an issue, so you can see which trunks are getting solid play buttons, and which ones are not getting recordings.  Be sure to filter out the calls that should be recorded by Event Talking, and only selecting the users or groups you are recording in Cradle to Grave.
      
      
       
      • NOTE:  If your phone system is Avaya IP Office 9.0 and older, be sure you have TFTP Configuration Read and TFTP Directory Read enabled in Avaya IP Manager so Chronicall can get relevant trunk information
   3. In the example below, once Trunks are displayed in Cradle to Grave next to the Recording column we can see that the calls on Trunk 9 are having issues
      • This step helps identify which VRTX is having an issue
        
        

Ensure VRTX Auto-Detect is Enabled

1. Log into Chronicall Desktop as an Administrator
2. Navigate to Admin (System) > Recording Libraries > (...)
   
   
   
   
3. Double-click the Recording Library in question, or select it and then select Edit
   
   
   
   
4. Set Enable VRTX auto-detect to True
   
   
   
   
   • NOTE:  If you have a VRTX Only RL, the setting is hidden so you cannot change it to True in the default installation. If you do not see the setting above, you may need to manually edit the settings.xml in the Chronicall install directory.  Be sure to make a backup of it before editing as you can break Chronicall modifying this file incorrectly
   • NOTE:  The following step should be done with all Chronicall services stopped.  It is recommended to make this change after hours so you don't impact normal call routing or call logging
   • If you do not see the setting above in Chronicall, please edit the settings.xml for Chronicall and manually change the value <recording.vrtx.enabled>false</recording.vrtx.enabled> from false to true, so it should look like <recording.vrtx.enabled>true</recording.vrtx.enabled> .  Save the changes to the settings.xml file and restart the Chronicall services and test.  

How To Check if You Are Running Out Of Recording Ports

1. Make sure there are no Out Of Ports folder in the Recording Library Directory, at the file paths below
   1. Default Recording Library file paths:

      • C:\Program Files (x86)\Xima Software\Chronicall\recording library 

      • C:\Program Files\Chronicall\recording library 

      • /var/lib/Xima\ Software/Chronicall/recording\ library 

2. If there is, the issue is that you are running out of recording ports, and would want to upgrade your licenses to accommodate the increased amount of calls meeting a recording rule
   • The total amount of your recording ports corresponds to how many recordings will be stored from any one time
     • E.g. If you have 8 recording ports, and 10 active calls that meet a recording rule, the last 2 will not show in Cradle to Grave.

VRTX Configuration

1. Verify the VRTX placement on your network
   • The VRTX is a line-side device and catches all VOIP events crossing that line that it has been told to capture via the Recording Rules
   • It can only see inbound and outbound traffic since it is monitoring the trunk.  Below are the two places the VRTX should be physically connected to on the trunk it is monitoring
   • PRI - Physical placement for the VRTX
     
     
   • SIP - Physical placement for the VRTX
     
     
2. Verify that the Windows Device Manager sees all of your VRTX devices
   • If you can see it in the Device Manager as shown below, you know that Windows is seeing the VRTX device and that the drivers are installed
   • If you do not see it listed here, then either there is no VRTX device physically connected or the device will need to be reseated within its USB (unplug and plug back into the computer).    
     • If you are on a virtual machine, this could indicate the USB's are not properly mapped
       
        
   • Restarting the RL service is also a good step to refresh the Device Manager
3. If you have multiple VRTXs on one machine and are unsure what VRTX is being referenced:
   1. Right-click the VRTX device, and then go to Properties for the VRTX Device
   2. Select Manage Identify > Device Instance Path to show the serial for that VRTX device
4. While here, also verify that all the USB Root Hubs do not have the ability to power down USB devices
   1. Right-click on each of the USB Root Hubs listed in Device Manager and select Properties
   2. Inside the Properties menu, select Power Management and verify the local USB hub cannot turn off the VRTX device
      • This will make sure Windows cannot turn off the VRTX's, even if you switch the USB port the VRTX connects to
        
        
5. The next step will be to verify that Chronicall can see the VRTX devices
   1. Go to Admin (System) > Recording Libraries > VRTX Configuration
   2. Select the trunk in question, and select Edit
      
      
   3. This is where you can see the VRTX Serial in the drop-down box
      • If the drop-down box is blank, that means Chronicall is not detecting the VRTX
      • If Chronicall is not seeing the VRTX
        1. Close Chronicall, and unplug the USB for that VRTX, wait 30 seconds and plug the VRTX back in
        2. Now Restart the Recording Library Service
        3. Relaunch Chronicall and see if the VRTX is showing in the drop-down as shown below
           
           
   4. On the Trunk Configuration Screen, you can also use Chronicall to auto-detect the Channel Range if this is a PRI VRTX
      1. This is helpful if the database has had time to populate with channel data between an inbound and an outbound call
      2. Select Detect, and if it succeeds save any changes made
         • Disregard if it cannot detect the channel range and hit cancel. 
   5. If this is a SIP line with a SIP VRTX, determine how many licenses or simultaneous calls the SIP can handle, and set the channel range from 0 to the # of max calls
      • E.g. If the SIP was licensed for 60 channels, the channel range below would be 0 - 60
        
        
   6. Device ID differences by PBX
      • Avaya IP Office Configuration
        • IP Office will show the actual trunk number as shown below
          
          
      • Avaya CM Configuration
        • Select the proper Trunk ID (Trunk Group for CM)
          
          
      • Shoretel Configuration
        
        • Verify there are no other recording solutions in place.  Shoretel can only work with one recording solution at a time.

Verify Recording Library IP Address

1. If your Recording Library is installed on the Chronicall server, follow the next two steps below.
   1. Navigate to Admin(System) > System Settings > Recording Libraries and click on the ellipses
      
      
   2. Select the Recording Library that is locally installed, and ensure it has the Recording Library servers actual IP address, and not 127.0.0.1
      
      

Testing Our Changes

1. Make a test call to see if the VRTX is now working
   1. First, verify the agent you will place a test call with meets your recording rules
   2. Make sure the test call is an Inbound or Outbound call with an external number as either the calling or receiving party
      • Since the VRTX lies outside the phone system, the test phone call has to be an outbound or inbound call, and not an internal call
   3. Chronicall should capture the recording in the Live folder
      •  Default Live directories 

        • C:\Program Files (x86)\Xima Software\Chronicall\recording library\live

        • C:\Program Files\Chronicall\recording library\live

        • /var/lib/Xima\ Software/Chronicall/recording library/live

   4. You can check here for any activity to see if the folder is getting the live data from the VRTX
   5. The Recording Library will then compress the recording into a .spx file and integrate it with the Recording Pool for storage after processing
   6. Check Cradle to Grave to see if it is or isn't showing
      • If you are not seeing traffic here, it is likely a misplacement of the VRTX
        • Refer to the VRTX Configuration section above to verify the VRTX location
   7. You can navigate to the Recording Library directory at one of the paths below, to view the most recent Recording Library logs to check for any related errors such as a Pack Splitter Lost Error or ALAW/ULAW RTP Frames received out of order or lost error (missed network packets).
      • 32-Bit Chronicall
        • C:\Program Files (x86)\Xima Software\Chronicall\recording library
        • C:\Program Files (x86)\Xima Software\Chronicall\recording library\logs
        • /var/lib/Xima\ Software\Chronicall\recording library
      • 64-Bit Chronicall
        • C:\Program Files\Chronicall\recording library\logs 
        • or C:\Program Files (x86)\Xima Software\Chronicall\recording library\logs
        • /var/lib/Xima\ Software/Chronicall/recording library/logs

VRTX is Falling Behind and Generating HS_ERR logs on Virtual Machine

1. To verify this, navigate to recording_library/bin/app in the Recording Library installation directory and open up a hs_err file
   
   
   
   
2. Look for an error message like below
   
   

   Current thread (0x000000002745f000):  JavaThread "processFTDI Thread" daemon [_thread_in_vm, id=8464, stack(0x000000002c1c0000,0x000000002c2c0000)]

3. If you see the error above the T1 manager is failing, meaning it cannot detect the devices
   • This indicates that the VM USB mapping is not working great
   • You can try disabling VRTX Auto Detect in the Chronicall settings, but this usually indicates an issue with the USB mapping and not the the VRTX
   • This is a workaround and not guaranteed to work, but the long-term solution would be to have the end user fix the USB mapping on the VM side