Bug fixed: Incorrect normalization of contract XML

A serious bug has been identified and fixed. It relates to the normalization procedure of XML using in smart contracts, and affects signatures and validation of signatures made before the fix was introduced.

Background

Before performing a signature, be it on a legal identity, or smart contract, the underlying XML representation of the object is first normalized. Normalization of XML means that the object can only be represented in one unique way, compared to general XML, where the same object can be represented in many different ways. For instance, in general XML, attributes are not ordered. In normalized XML, attributes have to be listed in alphabetical order. Normalization is important, to create a unique representation, so that digital signatures can be created and validated in a distributed environment. Since all participants generate the same object in the same way, they can compute signatures and validate signatures, and get expected results. (For detailed information on XML normalization, see Signatures in the Neuro-Foundation interface documentation.)

The Bug

The bug that was found consisted in invalid serialization of smart contracts, where the order of attributes were not serialized alphabetically in all cases. The incorrect serialization was however consistent across different implementations (due to reuse of serialization code), which made the problem invisible. Furthermore, unit tests that tested all smart contract operations, did not explicitly test that the XML normalization was performed correctly, so the bug went unnoticed, until a new implementation appeared that caused existing signing and validation operations to fail.

The Fix

The serialization code has been updated to conform with the normalization rules specification. Furthermore, all unit tests related to digital identities and smart contracts have been updated to include normalization tests, to assure proper serialization in the future. An analysis has been performed to see if this affects digital identies as well as smart contracts, but the bug seems at this point in time limited to smart contracts only, and the XMPP interface (including XMPP nugets).

Immediate Consequences

Following is a list of immediate consequences of this bug, when updating infrastructure to fixed code:

• Validating contract signatures across the fix boundary (i.e. using unfixed client with fixed neuron, or vice versa) cannot be done. Examples:

  • Partially signed contract with signatures from before the fix, cannot be signed with identities from updated clients.
  • Validating old existing contracts with new updated clients

Not affected

• Performing digital signatures using digital identities (a.k.a. quick login) still works, across fix boundaries.

• Petitioning and validating digital identities still work, regardless of version.

• State machines associated with tokens created with smart contracts signed before the fix. As the contracts was validates before the fix, the tokens and corresponding state machine have already been generated, and continue to operate.

• Applications that use the Agent API to create and sign contracts should continue to work (except as stated with contracts in general: signatures made after the update will be made with the correct serialization).

• Creating new contracts on new updated code, using templates created before the fix.

• Validating tokens generated using smart contracts signed before the fix, will continue to work. The token are valid and properly normalized before being created.

• Tokens created before the fix can still be transferred (sold) to other owners. Transfers are executed by signing new transfer contracts.

• You can still use the CreateContract script function to create contracts from templates created and approved before the fix.

• Validating new contracts based on templates approved before the fix.

#neuron, #bug, #fixed

Fixed Bug: Content-Only packages not detected properly

After updating the Neuron® hosting environment to .NET (core) 6 from .NET Core 3.1, it has been noted that Content-only packages are not detected properly. Previoysly, Content-Only packages could be installed in parallell (or automatically in parallell), without having to close the Neuron® during the installation. After the upgrade to .NET (core) 6, this feature stopped working, and Content-Only packages had to be installed as other packages containing assembly files, by temporarily shutting down the Neuron®. For packages such as the Neuron.Documentation.package, such an install could take considerable time, if Anti-virus software intercepted all file updates.

Cause

The detection mechanism in the Neuron® that pre-processed incoming packages to detect if they were content-only, or contained assembly files (.dll files), failed. The reason for this was differences of how the compressed and/or encrypted file streams worked between .NET Core 3.1 and .NET (core) 6. Reading a buffer (for example, 64 kB) full of bytes, would earlier return the entire buffer full of bytes, while in .NET 6, it could return a buffer less than full. This is properly handled in the external installation utility performing updates when the Neuron® is temporarily shut down. But it was not handled properly, in the method that checked incoming packages if they were content-only packages. This resulted in several content-only packages to be flagged, as not being content-only packages, requiring them to be installed when service is shut down.

Detection

If a content-only package is properly signed, uploaded, validated & detected for an update on a receiving Neuron®, and the Neuron® notifies its administrator that (here, the Neuron® being configured to automatically update content-only package-based installations; see the auto chat-admin command for this):

You can update the server with the new package from the chat interface.

Then, the Neuron® has failed to detect the package is content-only, and the Neuron® needs to be updated. The correct message should be:

The new software has been automatically installed.

Solution

Update the Neuron® to build-time 2024-01-26 (or later).

#bug, #fixed, #neuron

Bug Fixed: Problems with updates after update to .NET6

After updating the Neuron® hosting environment to .NET (core) 6 from .NET Core 3.1, it has been noted that further updates fail. This article describes the reason for this, and how to fix the problem.

Cause

The cause of the problem was twofold: The update procedure relied on classes related to cryptography and communication that has been obsoleted, but not removed, from .NET6, and shown not to work completely. Furthermore, there was a problem with linking to the System.IO.Compression assembly library, due to version conflicts. This caused incoming packages to be rejected as they could not be validated, effectively disabling the possibility to update, or automatically update the Neuron®.

Detection

When a new package for the Neuron® is distributed, events are logged showing the reception of the package, when the package is validated, if validation is successful or fails, and if a package is automatically installed. You can detect you experience this problem, if you see that the new package is received, but that validation of the new package fails, and the package is therefore thrown away.

Solution

If you experience this problem, that the Neuron® no longer can update itself, follow these steps to resolve the issue:

• In the \ProgramData\IoT Gateway\Packages folder, there’s a package called InstallUtility.zip. This file is already available on your Neuron®, in that folder.

• Extract the contents of that zip-file to the InstallUtility subfolder of the Neuron® installation folder. (Make sure files are not placed in a InstallUtility/InstallUtility folder. Default installation folder is C:\Program Files (x86)\Waher Data AB\IoT Gateway 1.0, and the InstallUtility folder would therefore be C:\Program Files (x86)\Waher Data AB\IoT Gateway 1.0\InstallUtility.

  • During startup, the Neuron® places the current installation tool in this folder. When the Neuron® updates itself, it assumes the intallation tool lies there. It can therefore call this utility to update its own files in the main installation filder, after shutting down, and therefore make sure its own files are not locked, and can therefore be updated. By placing the corrected files in this folder, makes it possible for the Neuron® to update itself, even though the files in the main folder still contains the bug that is to be fixed.

• Once the files are in the correct folder, make sure the most recent Neuron® package is downloaded, by executing the check command in the Chat Admin interface.

• When the latest package is available, execute the install or install nobackup command in the Chat Admin interface, as appropriate. If doing a backup, remember this process might take some time.

  • You can monitor the update process via the Task Manager as it happens. You will see the service stopping (after backup has completed, if that is the case), and the installation utility being executed in a separate process. Once the installation is complete, the Neuron® can be restarted again.

  • Note: Remember if you have the Neuron.Documentation.package package installed on the Neuron®, update time may be substantial, especially if you have an “antivirus”-program or defender-type program that intercepts each file begin updated.

#bug, #neuron, #fixed

Bug Fixed: Misspelling in State-Machine interface

A relatively consequential misspelling has been corrected in the State-Machine object model on the Neuron®. The property that holds the XML definition of the state-machine was mistakenly named XmlDefinnition. It has now been corrected to XmlDefinition, from build 2023-09-25 of the Neuron®. This change may have some consequences:

• Serialization and deserialization of state-machine objects during Loading and persistance is done automatically. In this process, the old, and now obsolete XmlDefinnition property is automatically transformed into the new XmlDefinition property, if this persistance or loading is done in a typed manner, i.e. knowing what type of object is being processed. This is the default case when processing state-machines and their events, in normal processing.

• The automatic conversion is not performed if untyped access to the state-machine object is performed. This can be done, for instance, when processing content using SQL or SPARQL queries, accessing the collection directly, without referencing specific types (i.e. using generic types).

Consequences

This change can have unintended consequences for all untypes access to the State-Machine object model. In the Neuron®, this only happens during backup, import or export of the database, or from third-party applications performing untyped access to the database. We are aware of no such third-party service that presents the state-machine XML definition, so no unintended consequences are expected.

Note: This misspelling does not affect the smart contracts and Neuro-Feature tokens creating the state-machine. It only relates to the internal object model for maintaining a running state-machine on the Neuron®.

#bug, #fixed, #state-machine

Bug Fixed: State machines did not persist the beforeActionScript attribute

State-machines using the beforeActionScript attribute on action reference nodes, such as event handlers, have lost their corresponding attribute values after restart. The reason is that the attribute value was not persisted.

Cause

The root cause of this is that the property was declared incorrectly. To be persisted, an attribute needs to have a public get and set method available, for the persistance layer to be able to persist it. Persistence is implicit, and no code is written explicitly to persist or load objects. Instead, the persistence layer relies on the declaration of the properties to persist. In this case, this particular property lacked such get and set methods.

Background

The beforeActionScript attribute was introduced to provide a means to re-use actions for different types of events. It allows the event handler to customize the execution of the action, by (for instance) providing variable declarations. (In such cases, the before-action script would work as assigning argument values in a function or method call in procedural or object-oriented programming languages.)

Tests

The problem was not detected in unit tests, since these are implemented in such a way to create new state machines for every test, to not rely on earlier execution, and therefore, they did not test more than the operation (which worked), and that persistance occurred without error. The tests missed that this particular attribute was lost once loaded after server restart.

Fix

The attribute is now declared properly, and value is persisted.

Review

After a review of the entire state machine model, it is concluded that this attribute was the only attribute that lacked a proper declaration.

Affected tokens

This problem affects generated tokens on Neurons with a build-time before 2023-09-21, if and only if they used the beforeActionScript attribute. After update, new tokens generated will persist the attribute correctly. Upgrading your Neuron® to a newer version, will not resture the attribute value.

Repair

A new method has been added to the State-machine object: ReparseDefinition(). As the original XML of the state-machine is available (and kept immutable in the token), the lost attribute can be regained by calling the ReparseDefinition() method on each state-machine, resave the objects, and then restart the Neuron®, for the changes to take effect.

Following is some script you can execute in the script prompt to re-parse all state-machines on the Neuron®. It will return a vector of all state-machine-IDs processed. Depending on the number of state-machines, complexities of the models, etc., execution of the script may take some time. (You can adapt this script if you want to apply it only to specific state-machines.) Make sure to take a backup before executing the script.

preview("Loading state-machines. This may take some time.");
Machines:=select * from StateMachines;
[foreach Machine in Machines do
(
	preview("Processing "+Machine.StateMachineId);
	Machine.ReparseDefinition();
	UpdateObject(Machine);
	Machine.StateMachineId
)]

Note: All changes to the operational state of the state-machine, as well as script executed via the script prompt, will be logged, both to the event log, as well as to the Ledger.

Note 2: If expecting many state-machines to be processed, you can use the TOP and OFFSET operands in the select statement, to paginate the processing into chunks.

Note 3: Keep the event log sniffer open while processing the script, to check for any unexpected events.

Note 4: Test this script in your lab/development environment first, and consider whether you need the update in your production environment or not, before applying the change. You can also modify the script to only process the state-machines you decide are important to update, to minimize any risks when performing the update.

#bug, #fixed, #neuro-feature, #state-machine