Status Updates

Weekly Development Report

CORE

A plan of action pertaining to the core rework was presented to a wider audience at the weekly technical meeting. The team made significant progress in researching various options in support of reverse engineering of the specifications as well as possible short-term performance improvements in the block storage layer.

Further progress was also made on the task of CBR-97 Machine-readable logging: specify format. A senior engineer has read the document and has provided some thoughts on how to progress. Senior engineers will have a conversation in order to decide how to progress this this task and what what needs to be done in order for implementation .

DAEDALUS

Last week the team received audit results for the Paper wallets certificate generator feature. The requested changes were implemented and tested and the feature is now ready to be released. The team disabled text selection on most of the static UI parts in order to achieve a more native look and feel of the application. As a part of the regular maintenance the team added multiple minor UI and code style improvements and extended automated test coverage by adding transaction display tests. Both the Daedalus and Cardano team members lists shown on the About dialog were updated on the scope of the preparations for the upcoming release. In the second part of the week the Daedalus team started manual testing and quality assurance tasks for the Daedalus 0.10.0 and Cardano 1.2.0 release.

WALLET BACKEND

Last week the team tackled various important tasks in order to improve exchange integration processes, streamline various wallet features as well as performance improvements. On the task of CBR-3 Decommission Cardano-SL-1.0- exchanges branch the team now awaits for an update from the exchanges to execute their migration plans, before moving further into decommissioning the old branch.

Furthermore, the team incorporated feedback received form a senior engineer, with regard to tracking metadata in the task of CBR-60: Write and validate formal specification for the new wallet backend. Also, the team provided the design of the acid-state DB for the wallet, combining the requirements of the formal wallet specification with the requirements of the Version 1 API designed by a team specialist.

Work continued on CBR-17: Implement wallet backend's new data layer following its formal spec. Last week, the team designed and put the acid-state layer into Github. Once this is merged it will also close four other tasks. Additional design work was done to match the Data Storage Interphase (DSI) -> Cardano Interpreter. The team reports that after a bug fix, which addressed previous slow performance issues, the active wallet generative tests are passing. The developers pushed to get the inductive rollback tests to run successfully. The problem still exists, however, the team successfully narrowed down the problem to how we are dealing with pending transactions.

Additionally, work continued on connecting the block listener to the passive wallet. The team was able to get the block listener to a stage where the it fills a channel with information about block events, which are read out by a worker process and applied to the passive wallet.

NETWORKING

The team mainly focused on CBR-96 Speed up verification and application of blocks. The team successfully added the ‘encodedSize’ function to the Bi class. The task of adding quick check tests is almost complete.

Also, significant progress was made on building benchmarks to measure which parts of the code could be improved as well as a quantifiable measure of those improvements.

More details about the network team's efforts are available in the CARDANO DECENTRALIZATION section of this report.

DEVOPS

DevOps spent much of last week supporting Daedalus and Cardano development and release planning. We identified a few scenarios where CI incorrectly succeeds, fixed immediate symptoms, and planned for longer-term improvements. Work also continued on the addition of integration tests to CI which spawn fresh Cardano clusters, claim Ada from the genesis block, and test basic wallet API functionality. Recent Daedalus configuration changes caused a regression in Daedalus development workflows, so we coordinated with Daedalus devs to fix it. More changes were scheduled for the upcoming Daedalus 1.2.0 release including security improvements for the new paper wallet feature, so DevOps built and internally announced RC-2. In support of API V1 stabilization efforts the V0 to V1 migration documentation was tested and improved, with a special focus on addressing the needs of existing exchanges. Additional work on Cardano and Daedalus include proposing configuration management changes to Cardano, helping developers benchmark block syncing, and discussing potential 3rd party Daedalus plugin mechanisms.

Cluster maintenance and cloud provider workflows also came up last week. Nodes on the internal staging cluster were nearing inode exhaustion. The team solved this issue by allocating larger volumes. We were also paged for a rare incident where a mainnet relay node failed status checks for a few minutes. The relay was fully operational after a clean reboot. Prompted by recent account management changes with our cloud provider, we have taken the opportunity to conduct an internal audit of how accounts and their API keys are managed and have planned improvements in this area.

DevOps worked with managers to ensure that both our team and teams that depend on us are productive. As part of the Praxis project we are keeping informed about QA's increased involvement with Cardano benchmarking and helping to facilitate the benchmarking team's handover. We have also been talking to other managers about creating cross-team development squads to diversify skillsets and improve focus on the most significant business objectives. Separately, we made plans to have external contractors provide the K-EVM and IELE testnet projects with more DevOps support, starting this week. Lastly, we discussed ways to improve and scale exchange engagement, and will be making a business proposal this week.

CARDANO DECENTRALIZATION

Delegation

Research paper - Vincent suggested re-evaluating the need for non-malleability, this might lead to simplifications in the address scheme. Apart from that, the paper is basically complete. Design document - Phillip made some additions.

Incentives

The main focus of the past week was the research paper. The experiments finally look very promising - stable behaviour that leads to a desired configuration. However, on the research side of things (i.e. the actual paper), the team still have to prove some theorems to align the theory with the experimental results.

Networking

Last week the team concentrated on various tasks. One of the tasks undertaken by the team was the effort to speed up block retrieval. A designated team member is working on testing and benchmarking the block streaming protocol, and ensuring it is backwards compatible. A different team member is working on block syncing micro-benchmark, which is aimed at measuring improvements and profiling as well as finding possible optimisations.

Also, a team member reviewed and merged a code backlog remaining after the 1.2.0 release was forked (they had been holding back some potentially disruptive changes from the immediate release).

Core Specification

'As is' State - Most of the rules that touch delegation have been specified.

MISCELLANEOUS WORKSTREAMS

K/IELE

Last week the team mainly concentrated on Runtime Verification development work, Mantis code merge and preparing for the internal testnet.

Marlowe

The main goal and time spent was on paper review, analyses and a presentation of the materials.

Smart Contract Language

The team focused on research topics, defining the roadmap and preparing a presentation plan for first testnet.

Reagan

Further testing was undertaken on a new AWS resource.

Tech Support

Last week saw an uptick in the number of tickets received. The tech support team received a total of 1066 tickets, out of which 270 were successfully. The top reported issues were:

• Unable to connect to network
• Unable to update wallet
• Cannot sync
• Misc.

Weekly Development Report

CORE

The team reports that further progress has been made on the task to finalize V1 API and test coverage for initial rollout. The team has successfully completed the task and addressed all issues pertaining to CSL-2398 - Wire up integration tests as part of CI (Continuous Integration)***. The developers plan to have the tests set up and running in CI this week.

In the case of CSL-2400 Collect missing V1 endpoint data compared to V0 endpoints the team has found that the wallet unit property does not seem to be used anywhere in deadfalls codebase, with the exception of one place where it is set to 0 without any modification. Per this finding, the team has concluded that the issue does not need to be addressed any further and plans to close it.

Furthermore, the developers expect that, from the design and implementation point, the task of CSL-2405 Add ‘feesIncluded’ option to Transactions endpoints will be completed in the current sprint.

Additionally, the team made headway on documenting additional common API use cases. Last week the developers extended the section pertaining to retrieving transactions history, in order to give more details about available filters. Also, rewording of schema definitions of ‘spendingPassword’ and various amounts to reflect more on their actual semantic, has been successfully tackled.

The team has also worked on writing a devops guide for exchanges. Initial version of the exchange documentation has been merged to the develop branch. This new version identifies many of the items previously requiring more definition and fill some proverbial “gaps”.

DAEDALUS

Implementation of the asynchronous wallet restoration was the team's primary objective last week. The feature has been successfully implemented, reviewed and tested. There are many UI changes introduced by this new feature: upon restoring dialog submission, users will straight away see the restoring wallet along with its balance in the UI. Wallet's entry in the sidebar now includes a linear progress indicator which shows restoration progress. On all of the screens of the restoring wallet, an active restoration notification is presented which contains both restoration progress percentage and estimated time of restoration completion. Asynchronous wallet restoration will be released in the next Daedalus release along with many other UI improvements such as the new ‘Add wallet’ screen and ‘Show more transactions’ button which is shown on the wallet's ‘Summary’ screen if there are more than five transactions in the given wallet. Support page within general settings has new and improved content. It now contains a link to FAQ section on Daedalus website, an option to open the ‘Support request’ dialog and an option to download the logs. NTP time check endpoint handling has been updated as this endpoint now captures both positive and negative time offsets. In order to improve security spending password setting is now switched on by default in both wallet create and restore dialogs. As a part of regular maintenance the team has updated Electron dependency.

WALLET BACKEND

In the past week the team undertook many important tasks and made some good progress in various areas. One of the tasks the team spent time on was the effort to decommission the Cardano-SL-1.0 exchanges branch.

A updated migration plan has been documented and fleshed out. As per the new plan, the flag to reflect how the previous stack-based build worked will be disabled. The plan is to start migration in the latter part of this week or early next week. The team currently awaits one of the exchanges to execute their previous migration plan. Another task undertaken by the team was the writing and validating formal specification for the new wallet backend.

After a review from a senior engineer, the history tracking section has been reworked. The section on the transaction submission layer has been rewritten. The changes are more precise and fit well with the rest of the spec. This part however still needs review by a senior engineer. Currently, the remaining and outstanding part of the spec is the study of the input section, on which significant progress has already been made by a senior engineer. Another part that requires attention is the acid-state database, which should be a relatively direct translation of the top-level functions in the wallet model.

Furthermore, in an effort to implement wallet backend's new data layer following its formal spec, a team member has made further progress on CSL-2382 - Implement “pure” data layer for passive wallet. A required testing suite now runs successfully, however, the speed at which it runs is too slow. Two team members have spent time debugging this together and realized that there was a function that was being called from the testing infrastructure too often. The testing infrastructure in the wallet has been extended to incorporate the fix. The pull request is raised and is located in GitHub.

Another team member reports progress on CSL-2381 - Extend ‘Inductive’ wallet generator to include rollback. Basic implementation is complete, however, the test suite is currently failing. The team will continue to push forward and take this task across the finish line.

NETWORKING

The main focus of the week was the effort to speed up verification and application of blocks. The developers added ’serializedSize’ calculation to the ‘Bi’ class. This should be used instead of re-serializing blocks/headers to compute their serialized size. Next step is to build a benchmark, 1) to measure how this helps, and 2) to run the benchmark with profiling information, and analyze it, to find other places where we can make the verification faster.

More details about the network team's efforts are available in the CARDANO DECENTRALIZATION section of this report.

DEVOPS

Last Friday DevOps cut the release/1.2.0 branch for Cardano and release/0.10.0 branch for Daedalus, and started preparing Daedalus 1.2 test installers for QA. Throughout the week, leading up to the release cutoff, the team focused on Cardano-SL and Daedalus concerns for the upcoming 1.2 release. While testing the API a team member identified a regression in wallet restoration. The bug has since been resolved by the new async restore feature. Automation tooling was updated to handle Daedalus pinning Cardano-SL by revision instead of branch, the versioning strategies in Cardano-SL and Daedalus were analyzed and a more coherent approach was proposed. To ensure the Linux installer can be included in the conventional release and update workflow, an engineer verified the blockchain-based update mechanism supports adding new tags. Following the Linux Beta release the community pointed out inode resource limits can be an issue in some context. In response an engineer prepared a Linux-specific proof of concept implementation for more efficient block storage. Initial findings are promising, and has motivated discussion about the proper cross-platform solution. Meanwhile, the recommendation for Linux users hitting inode limits is to use an ext3/ext4 partition with more inodes (allocating a larger partition generally helps) or use another filesystem that works well with many small files. DevOps also worked on a mix of support tasks, including helping developers with CI, cleaning up old clusters, and helping exchanges. To improve developer experience on GitHub and community visibility we switched the default branches for Cardano-SL and Daedalus from master to develop. The benchmarking team is preparing to get QA more involved in their workflow, and DevOps will be advising how to automate routine experiments. To get familiar with the current situation, an engineer reviewed the benchmarking team's training materials and reviewed a proposed plan for QA's onboarding.

CARDANO DECENTRALIZATION

Delegation

The researchers involved with the delegation research paper, discussed about quick recovery and shorter addresses. After which a special section pertaining to these subjects was presented to the team and was then reviewed. Initial feedback was good. The research team is reviewing the document further and might amend the paper slightly. Paper in its entirety is nearing completion. Additionally, the Hamburg presentation of the delegation research was well received and the researchers were able to obtain good feedback from the audience.

Incentives

A draft of the research paper is still in progress. The research team have agreed on a new set of experiments/simulations that a senior researcher will perform.

Networking

In their quest to understand and improve the block verification pipeline and speed up block retrieval, the team has been making tests for serialization instances, and looking into a more efficient way to deal with byte limits for blockchain data.

In regard to block streaming, a team specialist is still testing and benchmarking the streaming implementation, but has now turned to ensuring backwards compatibility with existing nodes that run old software.

On the subject of peer discovery, a team member has abstracted the networking provider so that the team can better test and simulate the system. The team feels that they could use this to gain confidence that it is resistant to various known attacks (the original Kademlia implementation is quite vulnerable).

Core Specification

Attention focused towards the current delegation mechanism of the 'As is' state, to prepare transition towards new delegation mechanism. A specialist has joined the effort of extracting the spec for the current system.

MISCELLANEOUS WORKSTREAMS

Last week the testing for KCC was ongoing on large AWS infrastructure. The team expects to complete 5M tests in under 40 days. Additionally, the team members have started work on blogs and paper. Testing of internal testnet for K-EVM is currently ongoing. The team reports that the delays are due to insufficient knowledge of the AWS infrastructure and performing the related DevOps role in the team. Also, the team engaged in supporting RV in their development and merging of code changes for automated faucet and wallet client for testnets.

Tech Support

Last week saw a small decrease in the number of tickets received. The tech support team received a total of 554 tickets, out of which 251 were successfully resolved and the remaining 303 are being followed up. The top reported issues were:

• Unable to connect to network
• Unable to update wallet
• Cannot sync
• Misc.

Weekly Development Report

CORE

Last week the the team engaged in a discussion in order to come up with an estimate for a refactoring plan for Core code base. The team feels that the most optimal approach is to have a minimal, viable refactor so that the team can determine and ensure that the Core is stable yet reasonable enough for for the team to be able to start opening up the core base to new features. Currently, the issue seems to be that the codes current state makes it difficult to optimally scope out the problem, therefore as a first stage a core stabilisation document will be produced, which will help estimate the work involved. The team estimates that this entire task should take less than two weeks. A senior engineer has provided some feedback for the document already, thus in the coming week the the team plans to complete the document and have a roadmap to move forward with the agreed approach.

• Exchange Onboarding & Support

The team reports that the documentation of Version 1 API has been completed. New use cases have been added to the documentation and are currently under review. Additionally, the team was able to successfully port over all endpoints required by exchanges to V1. Currently, the V1 API test coverage is being finalised. DevOps guide for exchanges is still in progress. As part of increased attention to optimising communication workflows between the development teams and the exchanges the team has started creating and sending out newsletters and created new Slack channels. Discussions are now under way to determine the an optimal exchange onboarding strategy for DevOps.

• Version 1 API Development

The team engaged in finalising V1 API and test coverage for initial rollout. In the task of (CSL-2375) Wallet API needs to be reviewed after Network Time Protocol refactoring, the new design is extensible for further changes to how we represent time to clients. Work is also underway on (CSL_2398) wiring up integration tests as part of continuous integration (CI). The primary work on integrating various parts has been completed successfully. Work involving (CSL_2400) collecting missing V1 endpoint data compared to V0 endpoints, needs to be updated to current state of the wallet. However, most of the changes have been addressed. Additionally, there is a possibility that (CSL_2405) Adding ‘feesIncluded’ option to Transactions endpoints, will be completed from a design and implementation point, in this sprint Furthermore, last week the team labored to review and gather feedback on the current V1 API. The team is currently implementing the changes which are under final review and awaiting final feedback. Another task which was concentrated on last week was to document additional common API use cases. The developers pushed to extend the section about retrieving transactions history to give more details about available filters. This included an effort to reword schema definitions of ‘spendingPassword’ and various amounts to reflect more on their actual semantic. Also, the team spent time on creating a log file analyser for bug classification. The collaborative team created an interactive project for the ticket classifier, which makes grabbing and managing tickets from multiple sources really easy. Last, but certainly not least, the developers worked to improve block syncing. More specifically the the team concentrated on bug fixing & performance improvements. The team reported that they successfully implemented this feature as well as a test setup to compare streaming against the existing batch download mode. Preliminary results show a roughly 1/3 speedup (streaming takes 2/3 the time) and memory/Garbage Collection (GC) stats are much better. During the current sprint the team will continue to refine streaming to maximise performance.

DAEDALUS

Last week the team finished the paper wallet certificate generator feature. The code was reviewed, the feature was thoroughly tested and the audit process is under way. The team participated in the QA testing phase and the preparation of the Daedalus 0.9.1 and Cardano 1.1.1 hotfix release. There is a substantial list of UI/UX improvements and minor bug fixes which were resolved and will be released with the next release. The team also started with the development of the next big feature which is the asynchronous wallet restoration.

WALLET BACKEND

The team concentrated on decommissioning the Cardano-SL-1.0- exchanges branch. There were unexpected errors in the previous migration plan, which was tracked down with the with-explorer. Apparently, a compile time flag unintentionally enabled in the new environment. An updated migration plan has been documented. This will disable the flag to reflect how the previous stack-based build worked. The team plans to start migration in the latter part of this week or early next week. The team also worked to write and validate formal specification for the new wallet backend. Further progress on the specification has been made. All the aspects of the rollback have now been documented in detail, definitions and corresponding proofs are also much cleaner. A senior engineer has done a first review of the new sections. Additionally, a section has been added for tracking metadata, which is needed for history tracking. However, this has only just been completed and is pending review at. In an effort to implement wallet backend's new data layer following its formal spec: Further progress has been made on (CSL-2382) - Implementing “pure” data layer for passive wallet, despite that the pull request (PR) is not ready yet. However, the design is ready, tests are hooked up to the generator. Currently, tests are failing and the team is debugging the issue. A team member has started looking at the new wallet from the point of view of running it in an actual network. He discovered some strange values reported from the server, and has got to the bottom of that. In the task of adding a server side Transport Layer Security (TLS) check, the Development / Testing certificates in scripts/tls-files have been updated, and a client certificate has been added for testing. The servers-side checks now include client's name verification (information embedded in the client's certificate, matching against values provided to cardano-node via the Command Line Interface (CLI), default to "Daedalus Wallet”). Certificates are generated from a script located on the Daedalus repository. This script has been slightly adjusted and a PR will be issued soon to cover the changes. End-user documentation extension has been submitted to describe the required manipulation of generating and using new certificates. Also, a pull request has been opened for the task of asynchronous restoration of a wallet from seed. This PR will not need to be reviewed by the development team. Front end team work is also required on this PR. Finally, the task of merging remedial work from the master to develop branch has been successfully completed.

NETWORKING

The team spent time on speeding up verification and application of the ‘blocksContinued’ analysis and found out that there are some aspects that need to be changed / completed. A change is required for the Database query in order to stop querying the same data three times during verification. Additionally, it is necessary to push the test for block size in bytes out of verification of the already constructed data.

DEVOPS

DevOps delivered the Daedalus 1.1.1 update for Windows and macOS, as well as the Linux Beta installer. The team automated more of the Daedalus update proposal procedure, improved runtime configuration mechanisms, improved CI build sandboxing, and sent a patch upstream to nix-bundle to clean up temporary directories. In preparation for API V1 of the wallet we finished reviewing the new documentation for exchanges (docs for migrating from API V0 in progress) and automated nixops tests were added to detect regressions. We reviewed and discussed plans to move toward structured logging, started improving fallback communication guidelines, and provided cloud resources for Project Reagan.

MISCELLANEOUS WORKSTREAMS

• Smart Contract Language

Last week the smart contract language team pushed to develop a roadmap and detailed plan for the first month of development and started an investigation into the feasibility of various data type encodings for the smart contract language core.

• K/IELE

The team is currently preparing for a code merge and an internal testnet.

• Misc.

The asset locality paper progress has been good. A draft should be available for review shortly. Additionally, we have started testing KCC (K-framework) on large AWS server.

Tech Support

Last week saw a small decrease in the number of tickets received. The tech support team received a total of 705 tickets, out of which 219 were successfully resolved and the remaining 486 are being followed up. The top reported issues were:

• Unable to connect to network
• Unable to update wallet
• Cannot sync
• Misc.

Weekly Development Report

CORE

• Exchange Onboarding & Support

Documentation of Version 1 API has been completed. New use cases have been added to the documentation and are under review. Port of all required by exchanges endpoints to V1 is done. V1 API test coverage is being finalised. DevOps guide for exchanges is still in progress. For exchanges, new slack channels were created. Discussions underway for how DevOps is going to onboard exchanges.

• Finalise Version 1 API and test coverage for initial rollout

CSL-2375 Wallet API needs to be reviewed after Network Time Protocol refactoring. The new design is extensible for further changes to how we represent time to clients. CSL_2398 Wire up integration tests as part of Continuous Integration. This work is well underway, the primary work on integrating various parts is complete. CSL_2400 Collect missing V1 endpoint data compared to V0 endpoints. Needs to be updated to current state of the wallet, most of the changes have been addressed. *CSL_2405 Add ‘fees included’ option to Transactions endpoints. There is a possibility that this will be completed from a design and implementation point in this sprint.

DAEDALUS

Last week the team concentrated on the finalization of the paper wallet certificate generator. All modifications required by the updated specification have been implemented and the feature is in the final review stage. Once reviewed and approved it will be sent for an external audit.

WALLET BACKEND

Last week the team tackled the task of decommissioning the Cardano-SL-1.0- exchanges branch. There were unexpected errors in the previous migration plan, which was tracked down with the with-explorer compile time flag unintentionally enabled in the new environment. An updated migration plan has been documented, this will disable the flag to reflect how the previous stack-based build worked. The plan is to start migration in the latter part of this week or early next week. Also, the team worked on writing and validating formal specification for the new wallet backend. More specifically, the team worked to implement the wallet backend's new data layer following its formal spec. As mentioned previously it was thought initially that these two user stories could have been completed within the previous time lines. However, now that analysis has been done on these tickets it has been noted that this will take longer than initially anticipated. The high-level timeline for this particular piece of work is an additional two to three months starting from April. Furthermore, considerable work was done to implement the wallet backend's new data layer following its formal specification. Progress was made on tidying up the structure of this user story, which now has the appropriate sub tasks associated to it. Further progress has been made on CSL-2382 - to implement “pure” data layer for the passive wallet, however the pull request is not ready yet. Additionally, the team worked on adding a server-side Transport Layer Security (TLS) check. Development/Testing certificates in scripts/tls-files have been updated, and a client certificate has been added for testing. The servers-side checks now include client's name verification (information embedded in the client's certificate, matching against values provided to Cardano-node via the CLI, default to "Daedalus Wallet”). Certificates are generated from a script located on the Daedalus repository. This script has been slightly adjusted as described in comments below and a pull request will be issued soon to cover the changes. End user documentation extension has been submitted to describe the required manipulation of generating and using new certificates. The team also worked to investigate high (and recurrent) IO traffic in the wallet. While providing support to exchanges, the team felt that certain exchanges were running the wallet in a container, therefore contention occured between both inside and outside of the container. It turned out that there was an issue with one of the exchanges, there is evidence of disk I/o resource exhaustion, which is a first guess. This may cause the Cardano node to enter recovery. Asynchronous restoration of a wallet from seed was another task the team engaged in during the past week. CSL-2247 - Extend ‘WalletInfo’ state to report the restoration progress is complete and is ready for review and testing. After that, it's just a matter of finalising DB migrations and making sure API docs is correct. There is work required from the frontend team on this as well.

NETWORKING

Progress has been made on studding the block verification stack.

DEVOPS

DevOps continued testing and improving exchange onboarding documentation, in addition to supporting existing exchanges. For Daedalus, the team made a server-side fix to resolve a UX problem in the bug submission flow, improved their internal automation for proposing installer updates on the blockchain, and began preparations for the upcoming 1.1.1 hotfix release. The team onboarded the IELE and K-EVM developers with their cloud provider for future testnets, worked on Continuous Integration(CI) infrastructure maintenance, and have made progress on the effort to cross-compile Haskell projects to Windows using Nix.

MISCELLANEOUS WORKSTREAMS

• K/IELE

Good progress on K/IELE work and we are now starting to build the internal testnet for K-EVM; greater visibility on RV work.

• Misc.

Reagan research reviewed with stakeholders and a decision to perform additional testing was agreed to reach a statistically significant result (funding proposal dependent).

Tech Support

Last week saw an increase in the number of tickets received. The tech support team received a total of 825 tickets, out of which 187 were successfully resolved and the remaining 638 are being followed up. The top reported issues were:

• Unable to connect to network
• Unable to update wallet
• Cannot sync
• Misc.