Status Updates

Weekly Development Report

DAEDALUS

Wallet

Last week the team finished all of the tasks required for the Daedalus 0.11.1 and Cardano 1.3.1 release which will also include an important fix for an issue on Windows platform where Daedalus couldn't start if the Windows username contained non-ASCII characters.

The team is finishing the integration of the V1 Cardano wallet API. As an additional improvement related to this task, the team started with the integration of the React-Polymorph infinite scroll feature which will be used to browse through the list of wallet transactions.

An issue with ThemeProvider which caused problems with dynamic theming within the React-Polymorph framework has been identified and fixed. The team added Storybook story which demonstrate dynamic theme switcher example.

App Platform

Last week the team completed work on a state persistence service that stores a snapshot of the platform on each change, which is then restored on startup. This feature also serves as the groundwork for implementing multi-step undo/redo at a later stage. The team continued the work on the theme switching feature.

WALLET BACKEND

Last week the team concentrated on optimizing wallet synchronization and stability, implementing REST handlers for node info and node settings, as well as refining the database so that the acid-state would be strict all the way down.

Additionally, a team member completed the document relating the Wallet Spec to the Cardano codebase.

The developers finished all the entry points of TxMeta, fixed all fields and added some tests in addition the team worked on Safecopy by trying to find best ways to compute diffs between Maps and Sets, in a way that would allow the creation of a Checkpoint from a list of those diffs.

Furthermore, the team worked on pagination testing for address listing in the listAddresses HTTP endpoint.

DEVOPS

CI performance was a major topic this week: the cross-compilation work is nearing completion and its integration has already started. The overall CI performance situation is undergoing review and already performance metrics were being planned and partially introduced. A stream of work to simplify Goguen CI has been delivered. Daedalus has seen some DevOps attention with regards to the GPU safe mode, unicode-on-Windows issues and upcoming testnet integration fixes. Organizationally, we had some documentation consolidation work, releasing some of the previously-internal documentation that is useful for third parties willing to spin up their own Cardano cluster. The infrastructure is being extended with a new way of accessing internal services, which should allow more direct access to monitoring across the organization. Log delivery was scrutinized this week, with discussions around more direct intra-organisational access to both backend and end-user logs for non-production clusters, which should allow better efficiency at issue resolution by developers. Monitoring also received further attention, with the time-sensitive aspect of Cardano block diffusion being supported with plans of monitoring and implementation enhancements. The Plutus project continued to see preparatory work with github CI integration, test cluster deployment fixes and cross-team training. The testnet preparatory work included a report-server update, Explorer fixes and also a measure of cross-team training.

CARDANO DECENTRALIZATION

Research and Design

The team plans to soon release the Delegation e-print.

A senior researcher and engineer spent time doing the finishing touches on the delegation design document. In particular, they still need to describe the new rewards sharing mechanism in detail, double-check where they will need public keys, and where public key hashes are sufficient, and update the chapter on tracking the necessary state.

Development

Block storage work was significantly delayed due to slow CI build times and wallet integration tests that did not meet quality standards. The team is currently working on rewriting the tests to comply with the aforementioned standards.

Additionally, team members spent time working on core optimization and making the tests more reliable

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us as a Community Manager, Director of Product Management, Technical Writer, Open Source Manager, as well as several others. Please see the IOHK Careers page for more details.

Weekly Development Report

DAEDALUS

Wallet

Last week the team was focused on the preparation of the Daedalus 0.11.1 and Cardano 1.3.1 release. An additional fix for a problem where Daedalus was improperly started in case users used the "Pin to taskbar" option on Windows platform was implemented and tested.

The team is still working on the integration of V1 Cardano wallet API with only a couple more of V1 API endpoints to be integrated.

App Platform

In the scope of the work for the proof of concept 1, the team has finished the foundation for state persisting and better separation of layers in the codebase. The team has successfully updated test suite to cover new changes.

WALLET BACKEND

The team spent a significant amount of time working on high priority tasks in anticipation for next week's release. This included updates to the code, testing, and code optimization. Additionally, some team members concentrated on improving restoration and focused on recovering accounts and per-account balances during wallet restoration. Instead of recording a lot of chain metadata in the wallet checkpoints, the developers created a way to record the bare minimum and instead rely on the underlying node to get the rest of the info.

Other developers have completed the BlockMeta test work and started on a “Wallet Spec ~ Code” document that maps the wallet spec types and definitions to the exact locations in the new wallet code, along with some more details and context on how the spec and codebase relate.

Also, the team has closed several tasks regarding the Tx history endpoint and made some progress on the child task, for incoming Txs tracking. The developers also added some entry points for the TxMeta for newPending and Foreign Txs. The only thing missing around TxMeta is to write the tests and change the persistence of the db to be on disk.

NETWORKING

The team have validated the design assumptions for Delta Q Measurements Design, with a set of real-world experiments between cloud computing data centres worldwide. This gives the team greater confidence that the design should support high performance at world scale.

Additionally, the team members have implemented a prototype of the Communication Protocol Design which supports simulation in a different style to the approach proposed by a senior engineer. The prototype is a relatively simple first approximation: it's consumer-driven, allowing for the negotiation of an intersection point (so-called read pointer), the download of headers and blocks from oldest-to-newest, and the re-announcement of the tip by the producer (extension or fork, relative to the read pointer).

A team member spent some time getting familiar with the Kademlia papers, in regard to Peer Discovery Implementation and with the existing peer-to-peer code-base. They have since been working on expanding the testing harness to make it somewhat realistic, and addressing some problems that were revealed in the bootstrapping process.

DEVOPS

Devops continued preparatory work for the 1.3.1 release, including participation in Daedalus hotfix development -- in particular the GPU safe mode and the direct launch issue. In addition, the Linux Daedalus continued ramping up for its 1.4 release.

A Cardano explorer fix has been developed and deployed by DevOps.

CI & deployment tools improvements continued, both in terms of fundamental improvements, such as the Linux-to-Windows cross-compilation and inter-project resource consolidation and also short-term fixes.

The team worked on monitoring reliability, including preparations for a datadog agent upgrade.

The Plutus project support is a new thread of DevOps activity, and this week we saw Haddock and Hydra being worked on.

Cardano public testnet launch preparations included further pre-deployment planning, QA support and tooling improvements.

Another major theme was end-user report handling, which saw a lot of effort going into investigation and fixes in Zendesk interoperability, but also the integration with the upcoming testnet.

CARDANO DECENTRALIZATION

Research and Design

The team reports that the Delegation e-print of the research papers is soon to be released.

Two members of the research team spent their time doing the finishing touches on the Design document. In particular, they still need to describe the new rewards sharing mechanism in detail, double-check where they will need public keys, and where public key hashes are sufficient, and update the chapter on tracking the necessary state.

Development

The development team spent time working on block storage as well as core optimization. Several of the team members dedicated their week to working on address discrimination and improving file readability and maintainability.

GOGUEN

Languages and Virtual Machines

The Marlowe team has now optimized their processes and is using company standard for work tracking.

A newly hired specialist has started working with Plutus and Marlowe team on CI and wallet integration as well as supporting K/IELE.

End Users and External Developers

Last week a testnet technical support engineer has joined the team. The engineer will support all Goguen testnets (up to 10 including both public and private testnets).

Testnets

The team reports a third week of supporting IELE testnet in 'production'; no issues reported. Additionally, the team has been working on tasks for IELE testnet and maintainability changes (logging).

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us as a Community Manager, Director of Product Management, Technical Writer, Open Source Manager, as well as several others. Please see the IOHK Careers page for more details.

Weekly Development Report

DAEDALUS

Wallet

The team continued the work on the integration of V1 Cardano wallet API in order for Daedalus to be ready for depreciation of the V0 Cardano API. Most of the work has been done and the code changes are in review and testing stage.

In the scope of maintenance tasks the team fixed a bug within Storybook setup and improved Gulp build setup.

This week the team is focused on the preparation of the Daedalus 0.11.1 and Cardano 1.3.1 release which includes a lot of improvements and fixes such as an improvement of the paper wallet certificate QR code compatibility, a feature to restart Daedalus in safe mode and most importantly Daedalus build, which runs on the testnet network.

App Platform

The team continued the work on proof of concept 1, focused on permissions UI and sandboxing app state based on permissions provided by the user. This work required changes of the test suite to better describe the features and cover the changes with integration tests. In the scope of permissions UI implementation, the team added a dialog to request access from the user upon initialization.

The team is also working on app platform theme switcher implementation which will allow easy theme change across the apps.

WALLET BACKEND

The team spent the week wrapping up work on wallet-related endpoints and now have the whole family of wallet-related endpoints ported to the new data layer.

More work was done on extending the new data layer to support updates and wallet restoration as well as Ada redemption. While conducting this work, the team identified the need for extending the wallet with the notion of "foreign transactions" to implement redemption effectively.

A team member spent their time on writing proper safe-copy instances for the new data layer. This is very important as it will allow the use of a proper, persisted database rather than a transient in-memory one.

The team also completed part 2 of the extensions required for BlockMeta testing - allowing for multiple poor actor addresses for test poor actors and made use of this in BlockMeta scenario testing. Next up, a rebase of note and then to bring BlockMeta Testing to a close.

Other team members added tests that focus on Sqlite and other tests that use combinations of different Layer endpoints.

Also, the team worked on the task of wallet restoration.

NETWORKING

The team continued their work on the TCP Performance tests for Delta Q Measurements Design.

The developers have started work on implementations for both proposals of the Communication Protocol Design. A team specialist has made good progress with state machine description of his protocol and added free monad machinery in the second approach which was discussed in the previous weeks.

A team member extended the peer discovery test framework to deal with more realistic situations. The new tests revealed that the previous efforts did not properly assess the process of bringing up the relays "cold". The team has started thorough analysis of the newly acquired data.

DEVOPS

The IELE testnets received a number of deployments, in support of post-release improvements. The deployment process was improved, as part of preparation for future integration between Cardano and Mantis deployment infrastructures.

The Byron testnet had seen further pre-launch preparations, including automation improvements, faucet and Zendesk integrations. The team are well on track to support the coming release.

On August 8, 11:15 UTC, a Cardano mainnet update was confirmed, and accepted 12 hours later, carrying the Daedalus 0.11.0 release and Cardano 1.3.0. Further work has been performed to facilitate graduation of the Linux installers to Tier 1 level of support. Installer GPG signatures have been added to the website. Additionally, the installer distribution process is finally being transitioned to the use of CDN. End users should see significantly improved download times due to regional caching.

Preparatory Devops support work was started for the 1.3.1 release.

The exchange interaction process was undergoing refinement, to facilitate efficient onboarding and integration of the growing set of exchanges.

Continuous integration facilities received further work on cross-compilation, which would allow to speed up Windows builds, and significantly improve their reliability. Inter-project consolidation of resources received attention, in terms of Hydra deployment and servicing. Better coverage of the Cardano upgrade process is being worked on, through automation of hetero-versioned cluster deployments.

Implementation and further planning of DevOps organisational security and robustness improvements were major topics this week.

CARDANO DECENTRALIZATION

Research and Design

The delegation paper was submitted to a conference last week. The team will find out later in the year if the paper is accepted. Delegation eprint is expected to be released end of next week.

A number of revisions have been made to the delegation design document which now includes the rewards mechanism. This now needs to be reviewed internally this week.

Development

The developers spent time working on the changes to the block storage and an important issue that came up with the way the addresses were handled.

GOGUEN

Testnets

The team has completed their second week of supporting the IELE testnet in 'production' with no issues reported.

Further effort is being put into working on tasks for IELE testnet and maintainability changes (logging).

DevOps started work to help accelerate development of Plutus/Marlowe.

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us as a Front End developer, Exchange Experience Engineer/Coordinator, Software Test Engineer, Senior Events Manager, Testnet Community Manager, Development Experience Manager as well as several others. Please see the IOHK Careers page for more details.

Weekly Development Report

DAEDALUS

Wallet

The team continued the work on the integration of V1 Cardano wallet API in order for Daedalus to be ready for depreciation of the V0 Cardano API.

The team also prepared updates to the integration of the problem reporting feature with Zendesk support. For the upcoming Byron testnet, testnet version of Daedalus needs to submit more details in the problem report in order for the support team to distinguish testnet version of Daedalus from the mainnet one.

App Platform

The team continued building proof of concept #1, focused on persisting of application and platform state. The team fleshed out new service permission flow for allowing and revoking access to certain services for the applications.

WALLET BACKEND

Last week the team worked on implementing all the wallet-related endpoints which, when completed, will meet the specifications set out by exchanges.

A developer has continued work on support for updates for V1 Of the API. After close analysis it was determined that this particular effort is intertwined with the wallet restoration work. The team has since engaged in the effort of writing code to bridge the gaps between the two tasks. In particular the developers wrote an adaptor that gives access to more of the underlying node state, and a module that gives current and historical accepted update proposals on the blockchain and adds this information also to the wallet checkpoints. This will provide the starting point for restoration.

Another team member implemented the first of the extensions to blockMeta testing (introduced a DSL BlockMeta type and interpretation to Cardano BlockMeta, this allows for full blockMeta comparison in the tests, which was previously doing weaker comparisons between expected and actual BlockMeta). Currently, the team member is working on the 2nd extension: allowing for multiple poor actor addresses in the tests.

The developers also further concentrated on providing support for history tracking and queries. In particular, the developers dove into Sqlite optimisation needed for his task. The developers also rewrote some of the queries and DB indexes.

Furthermore, the team spent time working on combining their wallet restoration work with their checkpoint and chain brief work, moving ever closer to getting wallet restoration to work. The developers set up some test scenarios and experimental results using org-babel, and used that as the starting point for restoration.

NETWORKING

The team performed TCP performance measurements from across the globe in their effort in the task of Delta Q Measurements Design. The task is expected to be completed in 4 weeks.

The team have started developing implementations for both proposals of the Communication Protocol Design. Team members have made good progress with state machine description of the protocol and added free monad machinery.

Additionally, the team furthered their efforts in facilitating Peer Discovery Implementation and spent time working on expanding testing as well as begun to look into integration with cardano-sl.

DEVOPS

Both KEVM and IELE testnets received a series of deployments, with IELE subsequently graduating into a public launch on July 30. The week saw a number of fixes to the Solidity compiler and Remix IDE integration, as well as preparation for major monitoring improvements.

The Byron testnet is on course for the upcoming release and has now entered pre-launch testing phase, while also continuing to receive deployment infrastructure automation improvements and the faucet code being reviewed by the team for operational concerns.

The Cardano staging cluster saw the 1.3.0-RC6 staging deployment. Further work on UTxO fragmentation experiments was being conducted. Development scenarios were addressed by improved support for custom clusters built with Stack. Wallet integration tests were being debugged. Demo cluster setup procedure was prepared and documented. Clusters now also enjoy enhanced deployment monitoring, improving organisational awareness.

On the Daedalus side, the 0.11.0-RC6 wallet update was proposed on the staging blockchain. An effort to redesign the end-user log submission for increased robustness was initiated. Further work on Linux installers included proposal automation and Appimage support, as well as work on Daedalus icon set refresh.

This week a separate effort on planning of DevOps organisational security and robustness improvements was started. DDoS resilience improvements were being laid out for the KEVM/IELE infrastructure.

CI improvements continued this week -- we worked on a wide range of issues, and importantly on the Haskell cross-compilation from Linux to Windows, which should help us to significantly reduce overheads. A bug in ghc-pkg was worked on. We successfully completed a major migration of our entire infrastructure to the version 2 of Nix package manager. To facilitate better test coverage of Cardano upgrade scenarios, new machinery for hetero-versioned deployments was being designed and implemented.

CARDANO DECENTRALIZATION

Research and Design

The delegation Design document underwent several revisions to now include the rewards mechanism. The latest revision is due to be completed by the end of the week and sent for review.

Development

The team reports that refactoring is due to be complete by the end of this month.

The team is wrapping up the work by labouring on removing partial field accessors and adding golden/roundtrip tests as needed.

The developers spent their time working on reuniting orphan instances with their types or type classes.

A team member removed the Mockable type class, a typeclass which is a layer of indirection which allows code to be run in different monads without requiring changes to the monadic functions, as well as removing duplicate libraries left from the task of moving networking-related modules from infra to networking, which requires some extra testing so he is working on setting up a test network.

GOGUEN

Languages and Virtual Machines

Plutus & Marlowe development has been progressing at a steady pace , however, the team is now looking to allocate more resources to facilitate better progress.

Testnets

The team spent their first week of supporting IELE testnet in 'production'. Work has started on post-launch plan with comms and marketing and discussions about events and promotion.

Also, the team arrived at an alternative strategy for the technical support of testnets.

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us as a Front End developer, Exchange Experience Engineer/Coordinator, Software Test Engineer, Senior Events Manager, Testnet Community Manager, Development Experience Manager as well as several others. Please see the IOHK Careers page for more details.

Weekly Development Report

DAEDALUS

Wallet

Integration of the V1 API with Cardano Wallet continued as the team worked on first two packages of work which is split in a total of five packages.

The team also spent some time on improving the development workflow by switching from Stack to Nix for building the Cardano backend. The only issue with this approach was running the local Cardano cluster which was not working with Nix but is now fixed. This improves the workflow significantly and improves builds speeds since most of the build artifacts can be downloaded from binary cache made by the continuous integration system (CI). Two team members already switched to a new workflow and polished the instructions and documentation for building and running the backend with Nix. The instructions and scripts will soon be available to the broader development community in the GitHub repository.

App Platform

The team started building application infrastructure needed for the upcoming Proof of Concept #1 which is focused on persisting of application and platform state. A unit of work needed for persisting the application state was completed. The team also worked on theming support on the application platform level.

WALLET BACKEND

The team concentrated on porting virtually all the accounts-related endpoints end-to-end (from the Servant handlers to the kernel functions).

Two team members have continued working on extending pure data layer to support rollback. This week the developers concentrated on implementation and testing as well as some refactoring which was required to optimize performance.

Another team member has continued work in their effort of writing tests for block metadata. After some discussion with a specialist, the developers have decided to expand the effort in two directions: define DSL-level BlockMeta for richer comparisons with Cardano + support multiple addresses to test poor actors.

Furthermore, the team worked on their effort to optimize wallet restoration. The developers wrapped up porting the core of the old restoreWallet function to the wallet-new codebase.

Additional tests were created and run on the functionality which would allow to statically compute transaction size estimates.

NETWORKING

Last week the team worked on Delta Q Measurements Design and has developed a test program to take measurements.

Several of the team members have begun to converge on Communication Protocol Design and have now agreed on the same unidirectional producer-to-consumer models. The remaining point of which needs attention is the value (positive or negative) of a peer with a longer chain following that of a peer with a shorter chain.

A team specialist has spent most of their time getting familiar with the current peer discovery library for Peer Discovery Implementation. The team member is now working on expanding tests and has begun to look into integration with cardano-sl.

DEVOPS

On Monday DevOps helped launch the IELE Testnet and associated tools (https://testnet.iohkdev.io/goguen/iele/) which enable Solidity developers to have more confidence in the correctness and security of their smart contract code. Goguen infrastructure improvements are ongoing to improve deployment workflows and robustness.

DevOps reviewed and tested the Cardano Testnet cluster more in preparation for launch. We provided QA with documentation on how to request testnet Ada from the faucet and use Daedalus builds specialized for the cluster.

While testing for the upcoming Daedalus 0.11.0 release, QA discovered a performance bottleneck due to excessive logging. DevOps helped them adjust the configuration to reduce logging. More improvements were made to the AppImage-based Linux builds, and unexpected challenges like DNS lookup failures were discovered. The nix-bundle approach for Linux builds is still viable, however, and one of the two approaches will be considered for a future release.

Cardano cross-compilation on Linux for Windows continues to improve. The focus now is on converging the build pipelines for the cross-compilation branch and the develop branch. The implementation relies on newer packages, so Hydra builds for the branch have been configured to use Stackage LTS 12. To ensure parity between the Windows-based builds, the team is considering how to enable more tests on both build pipelines. Build speed was improved by removing a polling step. Nix and Haskell code in the branch were partially reviewed and refactored to follow project conventions.

To gain practical experience with wallet UtXO fragmentation, a DevOps engineer created a new cluster and implemented a program which intentionally fragments a wallet's UtXO distribution. The engineer also implemented a program to defragment the wallet and verified it works in extreme conditions.

CARDANO DECENTRALIZATION

Research and Design

Last week the research team completed the delegation paper. The design document is nearly complete as well. Currently, the team is discussing and finalizing how the rewards will be collected. Once this is fleshed out, the team plans to present the findings to top management in order to finalize the decision.

Additionally, the team has been working with a senior engineer on a detailed technical implementation plan for delegation.

The incentives research paper has been reviewed and finalized by senior researchers and submitted to a crypto conference for peer review.

Development

The refactoring was initially planned to be completed this week, however the team uncovered some additional opportunities that would provide significant payoff for the level of effort required to complete. The team plans to finish up all additional development in the next few weeks.

A developer spent time working on removing partial field accessors and adding golden/round trip tests as needed. Another developer has spent their time working on reuniting orphan instances with their types or type classes.

The team also concentrated on removing the Mockable type class as well as removing duplicate libraries which will require some extra testing. The team is currently setting up a testing network for this purpose.

Additionally, the team has been working on rearranging the package dependency graph, in order to have the existing functionality with a much less complex structure.

GOGUEN

Languages and Virtual Machines

The team is supervising the work on Plutus and Marlowe.

Testnets

The week was spent on delivering IELE testnet to specification on time.

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us as a Front End developer, Exchange Experience Engineer/Coordinator, Software Test Engineer, Senior Events Manager, Testnet Community Manager, Development Experience Manager as well as several others. Please see the IOHK Careers page for more details.