Status Updates

Weekly Development Report

DAEDALUS

Wallet

This week the team continued working on the implementation of UI designs for Shelley features and the work is now nearly done. The implementation of both the 'Delegation setup wizard' UI and the 'Delegation center' screen are in the final code review and testing phase and are expected to be finished at the beginning of the next week.

In the scope of regular maintenance tasks, the team improved the screen switching logic for all screens that are available from the application menu. The team has also added a couple of smaller UI fixes for the UTXO statistics screen, designed to ensure accurate visual representation of the UTXO distribution chart regardless of the Daedalus window size.

App Platform

This week the team completed the initial implementation of the smart contract backend state handling, performed research into an existing GraphQL-based blockchain service layer solution, developed a style guide for the decoupled libraries, and assessed the output of the generated Cardano wallet OpenAPI client. A proof of concept was created to assist with the upcoming backend changes within Daedalus, which applied to the Daedalus application platform service stack. Work also commenced on the Cardano Wallet Provider implementation into the Cardano JavaScript SDK.

BACKEND OPTIMIZATION

The wallet backend server is becoming increasingly expressive and transparent thanks to improved logging. The team has been learning from their experience with the old wallet backend, and the new version will provide a better experience for users, including the Technical Support Desk who often make use of logs in resolving tickets.

The team is also facing its first big challenge with Jörmungandr integration, which requires non-trivial changes to some internal architecture. However, they have already had some success in construction and signing transactions using new, Jormungandr-compatible formats, and verified that the implementation matched that from the Jormungandr command-line tool. The next step in the integration is end-to-end testing and getting all the different layers working together.

NETWORKING

The networking team has been working with the consensus team on the final integration of the consensus protocol with the networking protocols, successfully creating a working node by the end of the week. The team has also extended the node with a full node-to-client protocol which allows integration with a local client such as a wallet or blockchain explorer application. Work has also been progressing with DNS subscriptions. Finally, the team gave a workshop at https://monadic.party on our in-house session type framework, which was very well received.

DEVOPS

The team has created both Snappy and Chocolatey packages for Jormungandr, as well as a Homebrew tap for Cardano products called input-output-hk/cardano. The team has also started planning the creation of NixOS services and tests in the cardano-node repository. Finally, the team has begun testing Cardano SL 1.5.2, which contains an OBFT bugfix and minor wallet improvements for exchanges.

CARDANO DECENTRALIZATION

The next iteration of the executable specification is in progress, including both a revised update system design and crypto foundations. This executable specification will then be ready to convert into the real implementation. The team is planning focused sprint activities to complete this work in parallel to the Byron rewrite work.

The team has also continued this week with property-based testing, with new generators for the update system and some new high-level properties of the ledger being defined. Progress has also been made on the design for invalid chain generation using genetic algorithms, which will allow the team to take valid chains and mutate them to produce known failures.

Work on abstracting over the cryptographic primitives in the Shelley executable specification has also begun, which will allow the Shelley implementation to stay as close as possible to the executable specification making integration with consensus layer easier in the future.

Finally, the team has been working on testing the correctness of storing a new block on disk. This is a significant operation since it triggers the chain selection process: can we add this block to the end of our chain, or does it extend another fork that we should switch to? What if we add multiple blocks in parallel, do we still get the right outcome? Using property-based tests ensures that this functionality is correctly implemented.

GOGUEN

The Plutus team made updates to the extended UTXO specification based on some feedback from various sources. They also updated some readme and contributing files to improve usability and added functions for running traces with defaults. Edits were also made to the Plutus core specification to fix rules for abstractions and to add references for the compilation paper.

This week the Marlowe team continued work on an MVP of block-based Marlowe contracts on Meadow and the ability for users to save their work in a GitHub gist.

The Education team are working on reviews of some research papers and adding content to the smart contracts and welcome chapters of the Plutus ebook.

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us. Please see the IOHK Careers page for more details.

Weekly Development Report

DAEDALUS

Wallet

This week the team continued work on the implementation of the UI designs for Shelley features.

The implementation of the “Delegation setup” wizard UI is nearly done and the team has started the implementation of the “Delegation center” screen. Several minor UI/UX adjustments have been implemented on the “Stake Pools” screen. These were mostly related to the “Stake Pool” tooltip’s positioning and display logic.

In the scope of regular maintenance tasks, the team has improved the “React-Polymorph” numeric input component by simplifying the logic and reducing code complexity. The team is working on improvements of the screen switching logic for all the screens that are available from the application menu. These screens are shown in the form of an overlay which requires the team to prevent the display of two or more of these screens at the same time.

App Platform

The team completed the Ledger Nano S key manager in the Cardano JS SDK, which demonstrates the benefits of establishing an interface for application-specific implementations to be configured by the application. A Cardano interface was also introduced, decoupling the SDK from particular implementations of the Cardano primitives, and the Rust-derived bindings were implemented as the default. Work was also dedicated to state handling in the Smart Contract backend, which aligns with the existing repository design. A diff on the current and new state of the contract will be used to manage triggers. The team has also been contributing to an upcoming integration effort, and is investigating the use of generated clients using the OpenAPI specification.

BACKEND OPTIMIZATION

The team has put effort into making the new wallet backend server usable for Daedalus. This is an important milestone because we want to have Daedalus starting integration as soon as possible now that the http-bridgebackend is ready and stable. In the meantime, the team is making progress with the Jörmungandr integration and is extending the testing in this area too. The developers are now able to spin up a Jörmungandr node and interact with it to retrieve BFT blocks. The next major step in this area is the ability to submit signed transactions and is landing very soon on the new wallet backend!

NETWORKING

This week the network team worked on tx-submission protocols. There are two scenarios where they will be needed: a client submitting transactions to a local node (e.g. a wallet submitting transactions), and node-2-node tx-submission protocol which will carry transactions across the network. They have different threat models, and thus we decided to have two different protocols. We also progressed with a dns resolver implementation, which now includes tests in simulation. The developers decoupled consensus node initialization from networking, and cleaned up and simplified the integration of consensus node with real networking. Additionally, the team out carried an analysis for ΔQ measurements and extracted delays depending on block size.

DEVOPS

The team has created helper launch scripts with nix for running Jormungandr in the https://github.com/input-output-hk/jormungandr-nix repository. These scripts allow launching a cluster using Genesis Praos with multiple stake pools running in the same process. The team has also added GELF support to Jormungandr to support logging to a central server so IOHK can review logs of participants of the Shelley testnet launch. The team has added NixOS services and tested a tool that turns NixOS services into docker containers in a docker-compose file.

CARDANO DECENTRALIZATION

The team are expanding their property tests to the update mechanism, which allows to modify protocol parameters on-chain. Once those are complete the developers will be able to run conformance tests of the update system, that check executable spec against the concrete implementation. Expanding these property tests and finishing the conformance tests are the last parts of the Byron ledger work, other than integration. Once this is done the team will be moving people over to the Shelley concrete implementation!

The team have a working PoC of a federated extension of the framework to allow a process forward observables using a new backend TraceForwarder to a distant process that runs the new backend TraceAcceptor to insert these data into its local switchboard and further process them. This allows a slim integration of the framework in an application doing benchmarking, logging and monitoring without including all the special dependencies and offloading the processing to another, probably distant, process.

Another new backend has been implemented to forward log messages to Graylog, a tool used by devops to aggregate logs.

GOGUEN

The Plutus team worked on ways to track gas usage, and started to look at ways to deliver better executable contracts. They also looked at how to link to GitHub for commits within the Plutus Playground and how to decode values from Plutus Core back into Haskell.

This week the Marlowe team evaluated the MonadApp type and continued work on tests for the logic of the "Undo" button and a way to record the Meadow simulations.

The Education team are working on reviews of some research papers and adding content to the Plutus ebook.

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us. Please see the IOHK Careers page for more details.

Weekly Development Report

DAEDALUS

Wallet

This week the team implemented an alternative software update path for situations where Daedalus is unable to connect to or sync with the network, so users never receive an automatic update notification. In this scenario, Daedalus will check for a newer version and prompt the user to manually update by downloading and installing the latest version of Daedalus from the Daedalus website. This update will be released with the upcoming Cardano Wallet 1.5.2 version, which consists of Daedalus 0.13.2 and Cardano SL 3.0.2.

The team is still working on the implementation of UI designs for Shelley features. The implementation of the 'Current and previous epoch info' screen has been finalized, while the implementation of the 'Delegation setup wizard' UI is halfway done.

The Daedalus visual themes implementation has been fully refactored to simplify maintenance of existing themes, as well as the creation of new themes in the future. The new approach allows the possibility of a custom theme generator which Daedalus users could use to create and share their own UI themes.

The team has also added a lot of additional diagnostic data to the 'Network Status' screen, which has been renamed to the more appropriate 'Daedalus Diagnostic' screen. Newly added data includes information about the user's machine, platform, Daedalus and Cardano node process information, Cardano node API port, and Daedalus state directory path.

In the scope of regular maintenance tasks, the team has fixed a minor UI issue with truncated content in the Japanese interface of the 'Block Consolidation Status' screen when the Daedalus window is at minimum size. The 'React-Polymorph' UI framework library has also been updated to the latest available release, which includes a couple of minor UI improvements for positioning of select box dropdowns.

App Platform

This week the team made good progress on Ledger Nano S support in the Cardano JavaScript SDK, successfully implementing the Key Manager interface to pull public key derivations, and making a start on transaction construction for the signTransaction behavior. A Docker-based development stack for establishing a local Cardano provider was also defined, running the new cardano-wallet backend, http-bridge, and a cardano-sl cluster. Finally, the execution service within the smart contract backend was improved to use the newly defined types and the port mapper from the server module.

BACKEND OPTIMIZATION

The team worked hard over the past week to deliver the new database layer that will underpin the Shelley wallet backend. This work is now complete and comes with a robust test suite combining several techniques (property-based tests, unit tests, stress tests, and state-machine tests).

The team is also pushing to ensure that all of the existing integration test scenarios are sufficiently generic to run on a Rust node in BFT mode. This will be a major milestone for both projects and is expected to happen very soon.

Finally, the team is getting closer to delivering a working development environment for Daedalus, to allow the integration work to begin there too.

NETWORKING

The networking team has been working on the transaction submission protocol this week, creating simple clients and servers which are stripped versions of real applications and used in simulation during development. The team also created a wiki page about the Byron proxy which will be helpful for DevOps engineers, as well as for tracking which usage examples are already covered.

The team also prepared a pull request which integrates ouroboros-consensus and ouroboros-network, allowing a simple node to run with various flavors of Ouroboros consensus algorithms (such as BFT, PBFT, Praos) over a real network (TCP/IP). Work has also been progressing on DNS management, which will be integrated with this node.

Finally, the networking team has also been looking for data analysis tools to present findings clearly and intuitively, which can then be backed by models that are being developed. The team has also been working on the implementation of latency distributions, which are going to be used in simulations of the distributed peer selection algorithm.

DEVOPS

This week the team has deployed the new Prometheus-based monitoring stack to testnet and mainnet, which is an optional flag in the deployment scripts to set up a Prometheus monitoring server. The team is also looking at Nix tooling that can convert a NixOps deployment of services to a Docker compose file, to allow better cross-platform deployment for stake pools.

In other work, the team has added an abstraction layer to iohk-ops to enable the use of different backends ( such as Rust or Haskell). A public syslog server has also been set up so that that stake pool operators can provide their logs to help the team analyze performance and bugs reported by stake pool testers.

CARDANO DECENTRALIZATION

Now that the ledger has been fully integrated, engineers took a few days to tidy up loose ends: with the ledger in place, some design decisions are now easier to evaluate. The team has also begun writing some property tests for the storage layer, which needs to deal with some tricky problems.

Another week of property testing in the Cardano Ledger this week too. The team has been formalizing and testing lots of high-level properties of the system that should help catch interesting bugs. The full chain transition system is currently being tested with all features except the update mechanism. The team has also been working on the generators for the update system and will hopefully put them into use next week.

GOGUEN

The Plutus team started investigating the use of GraphQL instead of Swagger for contract endpoints within the Plutus Playground, as well as having a self-contained documentation site deployed in the Playground. They also worked on a revamp of the Plutus tutorial content, made various editing and consistency changes, amended the vesting example, removed some redundant tutorials, and looked at using AsciiDoc features within the text.

This week the Marlowe team introduced a MonadApp type which enables easier testing of the application logic, and they worked on improvements to Meadow parsing errors. They also started work on tests for the logic of the ‘Undo’ button and a way to record the Meadow simulations, so that the current state could be captured and added to the list of states in memory.

The Education team are preparing for their presentation to the executive team. The working group is busy building out the skeleton of the Plutus ebook.

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us. Please see the IOHK Careers page for more details.

Weekly Development Report

DAEDALUS

Wallet

This week the team continued working on the implementation of UI designs for Shelley features.

The implementation of the 'Staking pools' screen has been finalized, while the implementation of the 'Current and previous epoch info' screen and the 'Delegation setup wizard' UI is in progress.

The team has added an additional Daedalus log file that keeps track of which versions of Daedalus have been used with any given account. This file will help in debugging Daedalus update issues and will be primarily used by the Technical Support Desk.

In the scope of regular maintenance tasks, the team has fixed a minor UI issue with a truncated 'active restoration' message on the 'Add wallet' screen, as well as truncated addresses on the 'Receive' screen in the case where a user’s wallet has a spending password set.

A theme and language selector has been added to the Daedalus Storybook implementation. This allows QA engineers to manually test different themes and languages in Storybook itself, which is essential for testing the new Shelley screens which are not available in the Daedalus UI.

App Platform

This week the team completed a security assessment for the Chromium-based JavaScript execution engine, including the development of a test suite to cover resource consumption attacks, restriction of network access, isolation from the host, and memory isolation from other contracts running in the engine. A load test was also written to analyze the performance of the design, and the API server refactored to consolidate the service functionality with the contract HTTP interface using GraphQL.

In coordination with the Plutus and tools teams, progress was made on finalizing the compiled contract interface, determining the boundary for where the HTTP layer should be applied. Work on support for the first hardware wallet key manager in the Cardano SDK has also started and will continue into next week.

BACKEND OPTIMIZATION

More progress towards the integration with Jörmungandr was made this week. With a node running in the CI environment and a network layer skeleton, the team was able to talk to Jörmungandr and perform the first integration tests with the new node.

Meanwhile, the last phase of development and testing for the SQLite backend has begun, and the team is looking into benchmarking the now heavily tested implementation. Benchmarks also function as stress tests to see whether the new backend can survive a heavy workload and if any errors due to heavy load arise. The team has also achieved support for the new address format in Shelley, comparing our result with the Rust implementation and giving birth to a battle-tested library for doing bech32 encoding and decoding in Haskell.

NETWORKING

The networking team has been working on the transaction submission protocol, ensuring it works with the new mempool implementation. The team also extended the pipelined client in typed-protocols so that it can wait for messages when no outstanding requests are made, which turns out to be useful for the transaction submission protocol. After an extensive review of the design of the protocol, the team has concluded that it will meet safety criteria in adversarial scenarios, as well as allowing the submission of transactions in a manner which will maximize available bandwidth.

The Byron proxy, the networking demo, and the consensus demo are now all using the full network stack, including version negotiation and a multiplexing layer over a TCP connection. This is an important milestone since it is now possible to run a consensus node in real network conditions.

The team has also started to work on DNS-based connection management, which will be modeled soon. The GitHub README file has also been updated, a wiki page was written about the network demo, and finally, various parts of the design document have been updated.

DEVOPS

This week the team's focus has been on updating the proposed integration with ouroboros-network, as well as OBFT improvements that will aid in the transition to Shelley. A new repo has been created, jormungandr-nix, that will contain scripts for running stake pools. The team has also implemented an open-source VPN to be able to connect different parts of a NixOps deployment between bare metal machines and multiple cloud providers securely, without having to depend on cloud provider specific VPNs, security groups, and firewall rules.

CARDANO DECENTRALIZATION

This week the team has been mainly focused on property testing both the executable specification and the concrete implementation. Some positive progress has been made on generating chains for testing, and the team has uncovered some subtle bugs in a few of the new property tests. Work has also started on a significant performance optimization for the consensus layer. Sometimes the same transaction is run through the ledger rules more than once, and in that case it is safe to ignore some of the checks, so the team is adding a way of selecting different validation modes, with different amounts of work. This should see a reasonable boost in performance, since repeating some expensive cryptographic operations will be avoided.

Additionally, the team has adapted the code such that it can be verified with LiquidHaskell (LH), a language extension to annotate Haskell types with logical predicates (termed 'refinement types'). LH derives these refinements from Haskell source code, its own refinements of basic types, and from annotations in code comments. The refinements are translated to logic terms and fed into an SMT solver. It can test for totality (e.g. pattern matching) and termination of functions (normally by proofing that its first argument is monotonically decreasing). The team has identified a handful of locations in the code that will benefit from these refinements and the explicit 'proofs'. On this basis, it is now possible to express more complex invariants in these refinement types.

GOGUEN

This week the Plutus team worked on updates to the extended UTXO specification and the plutus-wallet-api, as well as some minor refactoring within the Plutus transaction functions. Support was added to the Plutus Playground for dynamic endpoints, which results in a new error type that is available for use. Changes were also made to ensure that contracts can feed information about the contract state back to the user interface.

This week the Marlowe team worked on some simplifications to the code, as well as refactoring the frontend Nix code where they removed duplicate Nix definitions between Meadow and Plutus Playground.

The Education team are preparing for their presentation to the executive team next week. The working group continued efforts on the Plutus ebook.

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us. Please see the IOHK Careers page for more details.