ChunkedList

A new type of collection is available in a new library: ChunkedList<T>. It was created as a result of searching for processes that could be optimized, and the realization of how poorly a LinkedList<T> performs on modern computers. At the same time, the List<T> structure, while often outperforming LinkedList<T> in many cases, is not desireable. The conclusion was to create a new type of collection that supports both the benefits of using a LinkedList<T>, with the performance of the List<T>, or, as shown below, outperforming both.

ChunkedList Performance

The ChunkedList<T> is a generic data structure that is optimized for performance. It combines features from the LinkedList<T> structure, with features of the List<T> structure.

The LinkedList<T> structure generates nodes for each element, and arranges them in a linked list. It is very straightforward to add and remove elements. But the structure requires a lot of memory overhead, especially if storing small elements. It is also inefficient to process, as elements are not stored closely together, not allowing processor caches to be utilized efficiently.

The List<T> structure on the other hand, stores all elements in a single array internally. This makes access much faster. But adding elements can require the internal buffer to be resized, which can be a costly operation. To avoid this, the size of the increase grows as elements are added (i.e. size is doubled).

The ChunkedList<T> combines the two structures. Instead of attempting to have a single internal array, which requires resizing, the list creates chunks, and links the chunks together in a linked list of chunks. This allows for elements within a chunk to be stored closely together, while avoiding the need to resize the internal buffer, avoiding the necessary copying of data each time that happens.

Following are some performance benchmarks done, comparing the performance of the ChunkedList<T> structure, with the LinkedList<T> and List<T> structures. The ChunkedList<T> structure has comparable interfaces, allowing it to fit in code that uses either of the LinkedList<T> or List<T> structures. You can find the code of the ChunkedList<T> structure in the IoTGateway repository. The same repository also contains unit tests and the following benchmarking tests. You can also use the collection via its nuget.

The following benchmarks were done on a Windows 11 Intel i9 laptop machine, in Release mode. Different performance results may occur on different types of machines, and in different build modes, and for different processors. The benchmarks tests are done first on a small number of elements, to test how the structure works for small number of elements, and then does the same thing for a larger set of elements. The times are then compared to compute a relative performance (in percent) between the ChunkedList<T> and the LinkedList<T> and List<T> structures.

For all graphs, the x-axis represents the number of elements, and the y-axis represents the relative performance, in percent. An additional black line parallel to the x-axis marks the 100% level, where the lists perform equally. If the colored graph lies above this line, the ChunkedList<T> structure performs better than the structure represented by the graph. If the colored graph lies below the line, the ChunkedList<T> structure performs worse. The hidden implementations explain the variations of the curves, as memory exceed internal caches, or buffers have to be resized. The graphs are colored as follows:

Adding elements (last)

Simple addition of elements (last in the list):

Enumerating elements using an enumerator

Enumerating elements in a list sequentially:

Enumerating elements using ForEach

Enumerating elements in a list sequentially using ForEach() (not available in LinkedList, so enumeration is performed using an enumerator):

Enumerating elements using ForEachChunk

Enumerating elements in a list sequentially using ForEachChunk(), giving the ChunkedList<T> an edge. (Not available in LinkedList, so enumeration is performed using an enumerator, or List, where enumeration is done using ForEach()):

Enumerating elements using nodes

The LinkedList<T> and ChunkedList<T> permit the enumeration of content using a linked list of node objects that point to each individual element (LinkedList<T>) or each chunk (ChunkedList<T>). This iteration is compared to the ForEach() method of List<T>:

Checking if list contains en element

The Contains() method is used to check if a list contains an element:

Removing elements randomly

Removing elements randomly using the Remove() method:

Removing first elements

Removing first elements using the RemoveFirst() method (or RemoveAt() for List):

Note: The ChunkedList<T> structure outperforms the LinkedList<T> structure, even for this type of linked-list operation.

Note 2: The List<T> structure quickly diverges in performance for this operation. It is not suitable for implementing a FIFO queue, for instance.

Removing last elements

Removing last elements using the RemoveLast() method (or RemoveAt() for List):

Note: The ChunkedList<T> structure outperforms the LinkedList<T> structure, even for this type of linked-list operation.

Adding elements (first)

Simple addition of elements first in the list (or using Insert() for List):

Note: The ChunkedList<T> structure outperforms the LinkedList<T> structure, even for this type of linked-list operation.

Note 2: The List<T> structure quickly diverges in performance for this operation. It is not suitable for implementing queues or priority queues, for instance, where you process items from either end of the list.

Index operations (get/set)

Index operations using this[Index] is not available in the LinkedList<T> structure, so the benchmark comparison is only done between the ChunkedList<T> and List<T> structures:

Finding elements

Finding elements uses the IndexOf method for for ChunkedList<T> and List<T>, and the Find method for LinkedList<T>:

Finding last elements

Finding last elements uses the LastIndexOf method for for ChunkedList<T> and List<T>, and the FindLast method for LinkedList<T>:

Removing items by index

Removing items from their indices (RemoveAt() method) is not available in LinkedList<T>:

Inserting items by index

Inserting items by index (Insert() method) is not available in LinkedList<T>:

Adding a range of items

Adding a range of items (AddRange() method) is not available in LinkedList<T>. Both List<T> and ChunkedList<T> structures are optimized for adding ranges originating from arrays. A benchmark of this operation follows first:

A second benchmark is performed when the range is an enumeration not based on an array:

Copying elements to an array

Copying elements to an array using the CopyTo() method:

Creating an array of elements

Creating an array containing the elements of a list, using the ToArray() method (CopyTo method for LinkedList):

#new, #library

FTP Server support

Support for FTP and FTPS in available in the Neuron® from build 2025-03-20. FTP is an old Internet technology, but with a lot of support in different software and libraries. By providing support for FTP it is now possible to:

• Provide broker account holders with File Storage using FTP.

• Provide developers and operators with a mechanism to copy, edit and manage files in a simpler way compared to using terminal prompts (chat admin), web pages or remote desktop connections.

• Ensure files have consistent access rights in the operating system of the Neuron®. When using Remote Desktop or similar technologies to access the Neuron® for the purpose of copying files to the Neuron®, these might be copied using different access privileges than other types of content files. This might cause problems, such as inability to access the files from the Neuron®, or inability to update them via packages. Files copied to the Neuron® via the FTP interface will use the Neuron’s access rights (Local Service by default) when copying the files, ensuring that the Neuron® will be able to update the files in the future, as part of normal package distributions.

Protocol reference

The FTP protocol is mainly defined in a set of RFC published by the IETF. There are also some custom commands that commonly available, but not standardized. The main RFCs implemented are:

• RFC 959: FILE TRANSFER PROTOCOL (FTP)
• RFC 2228: FTP Security Extensions
• RFC 2389: Feature negotiation mechanism for the File Transfer Protocol
• REF 2428: FTP Extensions for IPv6 and NATs
• REF 3659: Extensions to FTP
• REF 4217: Securing FTP with TLS
• REF 7151: File Transfer Protocol HOST Command for Virtual Hosts

There are some limitations in the implementations (see more details, below):

• All FTP access requires authentication (i.e. login, or use of mTLS)
• All authentication and transfer require TLS encryption.
• Only broker accounts (i.e. XMPP accounts, not administrative user accounts) with FTP access rights explicitly set have access to FTP.
• Passive mode is limited to pre-defined ports.

Types of Channels

FTP was created in a time when establishing socket connections was cheaper than implementing streaming or embedding files over a single socket. For this reason, FTP, even for simple tasks requires two or more socket connections between a client (the uploader, downloader or manager of files) and a server (the file server). One connection is made for commands\ - the control channel. All file transfers or lengthy replies (such as a directory listing) are sent over a separate socket - one separate socket per transfer - the data channels. The server needs to manage available ports and provide them, as they are needed, to the different clients.

In the first versions of FTP, the server connected back to the client for each transfer, i.e. each data channel. But as the Internet evolved, and Firewalls were introduced, it became apparent that such an infrastructure is not particularly useful (in spite of NAT). A new type of data channel was introduced: The passive data channel (where the first was considered active, from the server’s point of view). A passive data channel is created by the client connecting to the FTP server using a second connection, establishing a new data connection, on which the transmission of the lengthy response or the file can occur.

TCP/IP Ports

FTP traditionally uses TCP/IP port 21. This port starts unencrypted, but the Neuron® requires the client to switch to TLS before it can authenticate itself. This process is called explicit encryption in some clients. Apart from explicit encryption, there is also an implicit encryption that can be used, often referred to as FTPS (not to be confused with SFTP, which is something else). The default port for FTPS is 990. FTPS works much like HTTPS, in that TLS is used directly upon connecting, and not negotiated through the protocol.

The recommended method to transfer data is through passive data channels. The original definition of passive channels allows for the opening of any port for listening on the server. This is not a good approach, from a security point-of-view, as a very large range of ports must be opened in the firewall for this to work. The Neuron® insteads allows for a limited set of passive data channel ports to be defined (by default 50000 - 50009). The Neuron® then alternately provides access to these ports to the FTP clients that are connected. The Neuron® also ensures the correct client accesses the correct ports. It also ensures to delay access to these passive ports, in case all are used.

The ports used, must be defined defined in the Gateway.config file on the Neuron®. You can change, add or remove ports as you see fit. Every change to the gateway.config file however, requires the Neuron® to be restarted. Example of the Ports section of a Gateway.config file:

<Ports>
	<Port protocol="HTTP">80</Port>
	<Port protocol="HTTPS">443</Port>
	<Port protocol="XMPP.C2S">5222</Port>
	<Port protocol="XMPP.S2S">5269</Port>
	<Port protocol="SOCKS5">1080</Port>
	<Port protocol="SMTP">25</Port>
	<Port protocol="SMTP">587</Port>
	<Port protocol="FTP">21</Port>
	<Port protocol="FTPS">990</Port>
	<Port protocol="FTP.PassiveData">50000</Port>
	<Port protocol="FTP.PassiveData">50001</Port>
	<Port protocol="FTP.PassiveData">50002</Port>
	<Port protocol="FTP.PassiveData">50003</Port>
	<Port protocol="FTP.PassiveData">50004</Port>
	<Port protocol="FTP.PassiveData">50005</Port>
	<Port protocol="FTP.PassiveData">50006</Port>
	<Port protocol="FTP.PassiveData">50007</Port>
	<Port protocol="FTP.PassiveData">50008</Port>
	<Port protocol="FTP.PassiveData">50009</Port>
</Ports>

You can also edit the ports from the Admin menu, under Sources & Nodes:

Note that the ports you define here must also be opened in the Firewall, for incoming TCP/IP connections to be permitted. If you use Azure to host your Neuron®, a firewall entry might look as follows:

Encryption

TLS encryption is mandatory for accessing the Neuron® using FTP. The Neuron® supports both implicit and explicit encryption. The Neuron® also supports Mutual TLS. This means you need to enable Client Certificates for the corresponding port in Gateway.config, and use a valid certificate on the client side when connecting to the Neuron®. If there is an account with FTP access rights having an account name corresponding to the Common Name in the subject field of a valid certificate, there is no need to use a password to access the account. The client will be authenticated as soon as it has provided the user name to the Neuron® FTP server.

Security Note: The weakest link in FTP is the matching of Control Channel with corresponding Data Channels. In active mode (not suitable for Internet use due to firewalls), this problem is not as big, since a third party cannot attempt to hijack a data channel. In passive mode, the client connects to the server twice, first for the control channel where it sends the commands, and secondly for the data channel, where a file transfer will occur. Hypothetically, a third party can hijack this data channel and receive a file not aimed for it. The controls to limit this are basic: First, the data channel is only open for listening exactly after the command has been issued, and only for as long as to receive one connection. This typically generates a window counted in milliseconds. Secondly, the server matches the Remote IP Address of both connections, and closes any data channel connection attempt from another IP address. It is therefore very unlikely that the data channel can be hijacked, but it is possible. To protect against this, it is recommended FTP conections be made using Mutual TLS (mTLS). When mTLS is used, the server also ensures the client certificate used is valid in both connections, and use the same public key. With this added level of protection, FTP becomes very secure.

Account Access Privileges

To give a broker account access rights on a Neuron®, you need to edit the account from the administrative page. At the bottom of the account, you need to provide a Root Folder for the account. This root folder does not have to be unique for the account, but it must exist. You also define whether the account as no access rights (default), read access rights, write access rights or both read and write access rights. You can also provide an optional maximum storage, counted in number of bytes. If you don’t provide a value for the maximum storage, the client will have unlimited access (i.e. until storage runs out). Once the account has been updated, the client can use FTP for file storage, within the folder provided in the configuration (and any of its subfolders). Write-access rights gives the account the right to create new subfolders recursively within the folder provided.

Using an FTP Client

If you want to use the Neuron® for file storage or management from your desktop. you will need an FTP Client to access the Neuron®. FTP support in Windows is limited to unencrypted access, which is not permitted by the Neuron®. WinSCP is an FTP client that works well with the Neuron® and is commonly used on different platforms. When using WinSCP to connect, remember to use either explicit encryption on the FTP port (21 by default) or implicit encryption on the FTPS port (990 by default). SFTP is something else in WinSCP. You then connect, and once you are connected you can access the assigned folder, using the logical folder /.

Troubleshooting using sniffers

You can troubleshoot the FTP communication on the Neuron® in two ways: In the Program Data Folder, there exists an FTP subfolder, where communication logs will be stored in XML format, for 7 days. You can also create a web sniffer from the administrative page (under Communication/FTP Sniffer) that can be used to sniff on the communication for a maximum of an hour. The content of the files will not be displayed in these sniffers, only what connections are made, what commands are sent, and their responses.

#new, #neuron, #features, #api, #protocol, #ftp, #security, #mtls

Troubleshooting Stack Overflow and Out of Memory Errors

The Neuron® normally runs as a Windows Service using the limited Local Service user account. Any errors in the Neuron® or any of the applications hosted within it, are normally logged to the internal event log, which distributes events to registered event sinks. These may record them in files, internal databases, the Windows Event Log, or forward them to external sources, both for real-time monitoring or storage. Communications can also be monitoried and trouble-shooted using sniffers that can be temporarily attached to different communication channels. Any Exception occurring within the system can also be logged separately together with stack traces, and statistics of such is generated when the Neuron® restarts. But one type of Exception is particularly difficult to troubleshoot, especially for a Windows Service that runs invisibly on a server: Stack Overflow, Out-of-Memory, Access Violation Exceptions and other fatal exceptions can typically shut down the service immediately, without any possibility to record details about the event for troubleshooting. The only notification an operator may receive, is that the Neuron® restarts unexpectedly. After reviewing the backups folder, the operator may also notice the database has been repaired, since the Neuron® was shut down ungracefully (i.e. the process was killed). There are a couple of ways you can find out what causes such an error.

Remote Debugging

One way to find this error, is to attach to the Neuron® process on the server using Visual Studio Remote Debugging tools. Once attached to the process, the debugger can be configured to break when the Exception occurs, giving you a chance to examine the stack trace before the process is killed. There are difficulties with this approach however. One is that the code is probably optimized for Release, and difficult to Debug. You must also open ports, install the remote debugger, etc.

Running as a Console

Another way to troubleshoot the Exception, is to run the console version of the Neuron® in a Terminal Window or Command Prompt. The difficulty here, is to make sure the Terminal Window is using the correct user privileges to mimic the environment the Windows Service has. This can be done by downloading PSTools from Microsoft. It provides tools to access internal features in the Windows operating system. Once installed, you open a Terminal Window (Command Prompt) with administrative privileges. You go to the folder where PSTools is installed, and you type the following command:

psexec -i -s -u "NT AUTHORITY\LocalService" cmd.exe

This will open a new terminal window with the correct user privileges. Before starting the Neuron® in the new terminal window, you need to stop and disable the Windows Service, so it does not interfere. Then, in the new terminal window, go to the installation folder of the Neuron®, and type:

Waher.IoTGateway.Svc.exe -console

It uses the same executable file that is executed by the Windows Service, but the -console switch starts it in console mode. This will enable terminal output of the internal event log, and any system message that will appear. If a Stack Overflow, Out of Memory or Access Violation Exception occurs (or any other fatal type of Exception), this information will be output to the terminal window as the process is killed.

#tutorial, #troubleshooting

Enhanced QR Code API

From build 2025-03-07, there’s a new enhanced API for generating QR codes. It gives you more control over the parameters that control the appearance of the code. You can also use the API to generate QR codes suitable for print.

The source of the API is /QR/ on the domain your Neuron® runs at. The text after the /, and before any ? contains the URL Encoded text that gets encoded into the QR code. After the ? follows optional query parameters you can use to control the appearance of the code. If arguments are left out, their default values will be used. Following is a table of query parameters recognized by the QR Code API:

QR Code API query parameters
Parameter	Default Value	Description
w	400	Width of QR code, in pixels.
h	400	Height of QR code, in pixels.
fg	Black	Foreground color (dots) of the code. A special value of Theme can be used to select the text color of the current theme of the Neuron®
bg	White	Background color of the code. A special value of Theme can be used to select the background color of the current theme of the Neuron®
c	depends on URI scheme	Color of icon overlaid on the QR code. The icon depends on the URI scheme encoded into the code. Using the c parameter allows you to control the color of the icon.
mc	same as icon color	Foreground color of the markers in the QR code.
omc	blended between mc and fg	The Foreground color of the outer markers in the QR code.
ac	same as icon color	Foreground color of the alignment in the QR code.
oac	blended between mc and fg	The Foreground color of the outer alignment in the QR code.
q	1	Quality of the image. It is the number of points in each dimension calculated per pixel. Provides anti-aliasing of the QR code, improving the quality of the image.

Examples

Consider the following URL, of a legal identity: iotid:2be3be1f-0f8f-7e1b-0808-cc7844e9e732@legal.lab.tagroot.io. We can encode it into a simple QR code as follows:

We can add smoothness to the code, by adding ?q=2 to the code:

If we want the code to adapt to the current theme of the web page, we add &fg=Theme&bg=Theme for instance:

If we want to generate a code for a client displaying the page in black-mode for instance, we can control the colors using the various color arguments available. Example:

If we want to use it for print, we can add (for example) ?q=3&w=1200&h=1200:

---

This article is being written

#new, #features, #api, #neuron

Login-Auditor Exceptions

From build 2025-02-27 it is now possible to configure exceptions for endpoints in the Login Auditor that monitors all login attempts. The purpose of the Login Auditor is to detect malicious users and temporarily block them, and finally permanently block them, before they can gain access to the system by guessing user credentials.

The Login Auditor is configured in the Gateway.config file (also accessible under the Sources & Nodes and Gateway Configuration in the administration portal). The configuration consists of a sequence of growing intervals, and the number of allowed failed login attempts permitted from each remote endpoint. By creating an exception, you can provide another sequence of intervals and login attempt limits for those endpoints. If they are more permissible, they can be seen as white-listed to some extent.

Endpoint exceptions can be defined in three different ways:

• By specifying the IP addresses explicitly, one at a time.
• By specifying IP address ranges using CIDR format.
• By specifying domain names, if the remote endpoints connect to the Neuron® using mTLS.

#new, #features, #neuron, #security