Troubleshooting Stack Overflow and Out of Memory Errors
The Neuron® normally runs as a Windows Service using the limited Local Service user account. Any errors in the Neuron® or any of the applications hosted within it, are normally logged to the internal event log, which distributes events to registered event sinks. These may record them in files, internal databases, the Windows Event Log, or forward them to external sources, both for real-time monitoring or storage. Communications can also be monitoried and trouble-shooted using sniffers that can be temporarily attached to different communication channels. Any Exception occurring within the system can also be logged separately together with stack traces, and statistics of such is generated when the Neuron® restarts. But one type of Exception is particularly difficult to troubleshoot, especially for a Windows Service that runs invisibly on a server: Stack Overflow, Out-of-Memory, Access Violation Exceptions and other fatal exceptions can typically shut down the service immediately, without any possibility to record details about the event for troubleshooting. The only notification an operator may receive, is that the Neuron® restarts unexpectedly. After reviewing the backups folder, the operator may also notice the database has been repaired, since the Neuron® was shut down ungracefully (i.e. the process was killed). There are a couple of ways you can find out what causes such an error.
Remote Debugging
One way to find this error, is to attach to the Neuron® process on the server using Visual Studio Remote Debugging tools. Once attached to the process, the debugger can be configured to break when the Exception occurs, giving you a chance to examine the stack trace before the process is killed. There are difficulties with this approach however. One is that the code is probably optimized for Release, and difficult to Debug. You must also open ports, install the remote debugger, etc.
Running as a Console
Another way to troubleshoot the Exception, is to run the console version of the Neuron® in a Terminal Window or Command Prompt. The difficulty here, is to make sure the Terminal Window is using the correct user privileges to mimic the environment the Windows Service has. This can be done by downloading PSTools from Microsoft. It provides tools to access internal features in the Windows operating system. Once installed, you open a Terminal Window (Command Prompt) with administrative privileges. You go to the folder where PSTools is installed, and you type the following command:
psexec -i -s -u "NT AUTHORITY\LocalService" cmd.exe
This will open a new terminal window with the correct user privileges. Before starting the Neuron® in the new terminal window, you need to stop and disable the Windows Service, so it does not interfere. Then, in the new terminal window, go to the installation folder of the Neuron®, and type:
Waher.IoTGateway.Svc.exe -console
It uses the same executable file that is executed by the Windows Service, but the -console switch starts it in console mode. This will enable terminal output of the internal event log, and any system message that will appear. If a Stack Overflow, Out of Memory or Access Violation Exception occurs (or any other fatal type of Exception), this information will be output to the terminal window as the process is killed.
Fixing keyset errors on the neuron
If you have trouble setting up your neuron and see in the event logs, which you can see in IoT Gateway/Events/ get one of the errors:
- “Unable to get access to cryptographic key for database file C:\ProgramData\IoT Gateway\Data\<FILE_NAME>. (Object already exists.) Was the database created using another user?”
- “Unable to get access to cryptographic key for database file C:\ProgramData\IoT Gateway\Data\<FILE_NAME>. (Keyset does not exist.) Was the database created using another user?”
- “Unable to get access to cryptographic key for database file C:\ProgramData\IoT Gateway\Data\<FILE_NAME>. (Access denied.) Was the database created using another user?”
You installed the neuron in a bad way. What happened to me was that I had run the installer as administrator. This then creates the encryption keys, and then when the neuron starts running, it does not have the privileges to read them.
I recommend reading all the steps once before beginning the restoration. And always be careful, I figured out these steps on a computer that did not have any sensitive or important data so was not worried about breaking anything. Lastly, I would not consider myself deeply knowledgeable about the windows operating system so take everything with a grain of salt and preferably double check that what you are doing is correct.
What to do:
- Uninstall the neuron
- Delete everything in IoT Gateway/Data (be careful as this will delete all persistence files/data on the neuron)
- Go to ProgramData/Microsoft/Crypto/RSA/MachineKeys
- What we need to do here is to delete all the keys created by the installer, these keys are the persistence encryption keys generated by the installer. There probably are multiple ways of finding the correct files to delete but one way is to find every file containing any file path relating to /Iot Gateway. To do this download and extract the windows utility tools at https\://learn.microsoft.com/en-gb/sysinternals/downloads/psexec .
- Run PowerShell as administrator and navigate to the folder containing the tools.
- Run “./PsExec.exe -i -s C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe”. This will start a PowerShell instance running as the SYSTEM user (this is neccesary to later read the machine keyset files).
- Accept the terms and conditions of the external tool. Now you need to be careful since you now have a PowerShell logged in as the SYSTEM.
- Navigate to C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys
- Run the command: “findstr /s /m “IoT Gateway” /*” This will list all the files containing the string “IoT Gateway” in its content. You should be good to go to delete all those files (you can do that with the file explorer as a normal administrator) but be careful to not delete any other files. If you are comfortable, you could create a script to delete these files, but it should not take too long to do it manually, especially if you only have the Files.master file in /Data (every file there got its own encryption keyset). A few tips to speed up the process of deleting the keys are:
- The keys are usually concurrent when sorting by Date modified in the windows explorer
Whenever you see a long time step in date modified, look in both files to see if they are a part of the IoT Gateway keysets, (run Get-Content <filename> in the PowerShell running as SYSTEM) 
Or check that the files are in the file list outputted by the findstr
Then lastly reinstall the neuron as normal.
Now it should work as normal again.
Notes:
- These instructions could also be useful if you manually deleted all the files in /Data and when trying to reinstall the neuron get one of these errors.
- Other ways of finding out which files you should delete if you don’t want to use these external tools, you could find the keys created at the same time as you installed the neuron, and make yourself the owner of the file, give you read permission to see if they are the correct one (do this with the first and the last in the time period when sorting by date modified to find which “block” of keys where created by the installer). Though i would recommend doing the first way explained since it contains fewer risks regarding not potentially delete or changing permissions on unwanted files.
Troubleshooting
Tools on the Neuron to troubleshoot
The Neuron has multiple tools to troubleshoot depending on what communication channel you are troubleshooting.
Sniffers are only active for a relatively short time so as not to create accidental leaks. Sniffers should be open when testing is started, and only for legitimate reasons like debugging, development, or security analysis, and should be closed when testing is done.
Client Connections
Here you can monitor all client XMPP connections to the Neuron server. Especially helpful if you are working with the Neuro-Id™, Neuro-Access™, or any other applications using the Neuron®.
Events
On this page, you can follow events as they are logged in real-time. Always good to have open whenever testing.
FireBase Sniffer
On this page, you can follow the XMPP communication made from the machine to the Firebase service back-end.
HTTP Sniffer
On this page, you can follow the HTTP and HTTPS communication made from web clients to the machine.
Since the HTTP sniffer is HTTP-based, it can be used to create a loop if several individuals look at the same page or if there is some unsynchronization, thus enabling HTTP or web-socket traffic sniffing. This should only be used in a development environment.
Search Event Log
Search historical logs and filter with different attributes.
Server Connections
Here you can monitor all server XMPP connections to the Neuron server.
XMPP Sniffer
On this page, you can follow the XMPP communication made from the machine (as a client) to its (parent) broker.
Beyond the web sniffers.
In addition to web sniffers, various alternative tools are available for troubleshooting. Logs and communications are stored in distinct directories and subdirectories (under the IoT Gateway directory) for a limited duration, depending upon the configuration in the Gateway.config file. Access to the Neuron® server is necessary to review these records retrospectively. The following directories hold significance in this context and can be highlighted with brief descriptions:
- Events
- Exceptions (comprising both text files (logs) and XML files (statistics))
- HTTP
- SMTP
- UPnP
- XMPP (about neuron communication with the parent neuron)
XMPP_C2S(denoting client<->server communication)- XMPP-S2S (referring to server<->server communication)
Importantly, XML-based logs remain open until explicitly closed. These logs are assigned new filenames every hour, including an embedded XSLT file. This embedded file facilitates the conversion of XML content into HTML, making it viewable in web browsers. When reviewing unclosed XML logs, the process involves copying the file, manually adding the end tag using a text editor, and then opening the copied file in a browser to view its content.
Example:
(TODO)
Database Export
Moreover, a practical utility under the Backup section in the Admin dashboard permits users to export database content (or parts of it) or Neuro-Ledger® to a designated file. This export can be in the form of an XML file, which users can navigate through. It’s worth noting that the file size can be substantial if the entire content is exported, but the utility is handy for exporting targeted portions to facilitate troubleshooting.
Chat Admin & Script-Promt
Furthermore, the Chat Admin and Script-Prompt tools are important and can be very helpful in debugging. It’s essential to delve into the concepts of “Collections & Labels” and “Namespaces & Types.” (TODO)
For those who require debugging capabilities for script and Markdown pages, the “Lab” section in the Admin dashboard of the Neuron® offers several relevant options, including:
- GraphViz
- Markdown
- PlantUML
- Script (Prompt)
Posts tagged #troubleshooting
No more posts with the given tag could be found. You can go back to the main view by selecting Home in the menu above.