The Administration\Trace analyzer page is designed for capturing network traces in the PCAP format for troubleshooting of partner channels. Initially, this module was intended to give access to traces for our hosting platform users who do not have access to servers themselves, thus cannot run a trace-capturing command (tcpdump) directly from the server.
The System captures network traces automatically on a non-stop basis and by default stores them for three days (72 hours). The Trace analyzer page does not affect how traces are written down, it simply allows filtering the required file from the server that already had been captured automatically (that's why it is not possible to set a trace task "in the future"). So, filtering tasks can be created only for a past period not exceeding three days (default pcap storage period). However, the storage period can be changed by request:
- If you are our hosting client, you can contact your account manager to discuss the storage period extension and reach out to the technical support team once an agreement with the account manager has been achieved. Kindly note that increasing the period implies allocating more resources to the disk.
- If you are a non-hosting client, you can add sufficient space on the server and contact the support team to increase the trace storage time. The support team will advise you how much space you need on the server to provide the storage time you require. Traces are stored on the server in /var/log/invoice/pcap/ folder.
FOR INVOICE CLIENTS: by default, traces are only captured for SMS carriers, trace capturing for Voice carriers (if you have an inVoice System installed) is possible upon separate request to technical support. Also, Voice traces cannot be captured through Trace Analyzer if the traffic has gone through the SS7 module. The support team can capture such traces for you directly on the server or set up SS7-traces capturing on a regular basis.
Common issues with SMS trace capturing
1. No packets found within the requested period because the period is not set correctly, examples:
- You are trying to capture traces for the past period exceeding the trace storage period (72 hours by default), all traces exceeding the storage period are deleted, no archive traces are saved
- You're trying to capture traces for the exact current moment, which is not possible, since the Trace Analyzer handles traces not in real-time, but with a small delay, meaning that if you want to take traces for 15:00-15:15 with current time of 15:18 then you have to wait several minutes to run a task.
- You're trying to capture traces into the future that are not yet on the server (remember that the capture is done by System timezone, which you can see in the bottom right corner of the page).
2. Traces are TLS/SSL-encrypted and you cannot see the contents of the packets. Incoming TLS/SSL-encrypted traces cannot be decrypted. If you need to see how TLS/SSL-encrypted messages are sent to the vendor, outgoing TLS/SSL-encrypted traces can be captured only in real time directly from the server, catching them at the stage before they reach the stunnel System module that deals with SSL encryption. You can contact the support team to capture such traces, however keep in mind that only decrypted signals sent from the System will be visible in these traces, responses from the vendor will not be present.
3. Traces for VPN-partners are not displayed properly. In order to be able to see the correct packets exchange for VPN interconnections, it is needed to enable VPN traces capturing on the servers, which is done via the support team. There are two options:
- Enable key-dependent VPN traces capturing without a special deciphering module. This option is useful when there is no need to analyze VPN traces frequently and the configuration of VPN trace collection is needed quickly. Once the option is enabled by the support team, you will need to specify the Partner's Peer IP / GW IP in the Hostname parameter of the Trace Analyzer interface, ignoring the fact that it's not among all listed IP addresses in the dropdown menu. The task will still be executed for the inputted IP address. In order to decrypt such traces, you will need to request a special esp_sa key from the support team, there is a separate unique key for each VPN connection. Having the traces and the esp key, you can to decrypt them using Wireshark. The key application is described further in this article in the How to apply the esp_sa key in Wireshark section below.
- Enable decrypted VPN-traces capturing in the Trace Analyzer, no key required. The option is more time-consuming (an additional submodule will need to be installed by the support team), but it will help to view traces more conveniently without the need to use a key. Once the option is enabled by the support team, you will need to specify the Partner's Encryption domain in the Hostname field (not Peer IP, as in the first option). Encryption domains can be retrieved from the Host name field in the Carriers/SMS channels interface. In other words, traces for VPN channels will be captured the same way as for usual connections.
4. The port used for connection with the partner is excluded from trace capturing job. By default, we capture TCP traffic on port ranges 9000-65500, excluding port ranges 1-1024, 6379-6384, 7381-7384, 4000-4012 and excluding specific ports: 7788, 22031, 1521, 9200, 32803, 32769, 20049, 5000. If you need to capture traffic from one of the excluded ports, you can contact the support team so that this port is added into the configuration.
5. Trace-capturing tasks for long periods (longer than an hour) are not successful. Since trace exports are stored on the server with a limited amount of free space in the partition, running high-load tasks may cause the partition to overflow, which in turn will prevent Trace Analyzer from working. It is recommended to select periods of trace downloading not exceeding several minutes in order to exclude unnecessary packets and avoid overloading the partition. For example, you can check in the Analytics that the partner had traffic from 17:15 to 17:19, so there is no point in exporting the whole period from 17:00 to 18:00. It is advised to capture traces with a specific test SMS in mind for checking, so after finding out at what time the test SMS was sent, you can capture from 2 to 5 minutes around this time interval.
Other possible issues are more difficult to analyze on your own, so if you checked the information above and could not find a solution to why tasks in the Trace Analyzer interface hang, take a long time to load, are loaded empty despite the obvious presence of traffic or the files are not found despite the corresponding storage period is followed, you can contact the support team by providing Trace Analyzer TASK ID and describing the issue in detail.
How to apply the esp_sa key in Wireshark
In order to apply the esp_sa key in Wireshark on Windows, the esp_sa file must be placed in C:\Users\%your_account_name_here%\AppData\Roaming\Wireshark\ as esp_sa (i.e. as it is, without changing the name). The AppData folder can be hidden on your computer. You can locate and allow hidden folders to be seen according to the instructions in the following link.
Once the key has been placed in the appropriate folder on your computer, make sure that Wireshark is configured to apply keys automatically. In particular, the following settings should be set in the Wireshark interface (Edit ⇾ Preferences ⇾ Advanced):
esp.enable_null_encryption_decode_heuristic: TRUE
esp.enable_encryption_decode: TRUE
These settings can be found via the following path: Edit ⇾ Preferences ⇾ Advanced. Below you can find screenshots of the English language interface for reference, taken from the official Wireshark website and not reflecting your specific trace:
1. the Edit menu with the Preferences menu item, example:
2. the Advanced menu with the Search field, example:
You can use the Search field to find the two settings mentioned above and set them to TRUE if they are not already.
Once the key is installed in the folder and these two settings are applied, you can open the .pcap file itself, the VPN traces will be in a decrypted form.
AKBSMS - Alaris Knowledge Base
Trace Analyzer issue.
Trace issue.
Cannot capture traces.
Empty files in Trace Analyzer.
No packets are found in Trace Analyzer.
How to change trace storage period.
Capturing traces for VPN connections.
TLS-encrypted traces.
How to apply the esp_sa key.
Link to this Article: https://helpdesk.alarislabs.com/en/knowledge_base/article/253/category/131/