Status Updates

July, 2018

Jump to

July, 2018

July 26, 2018

Weekly Development Report

DAEDALUS

Wallet

React Polymorph user interface library was expanded with a progress bar component and components for managing layouts.

For the rest of the time, the team mostly worked on integrating Cardano wallet API v1 with Daedalus, since that is one of the main features for the upcoming Cardano 1.4 release.

App Platform

The team has decided to build a series of proof of concept releases to address upcoming key features for the application platform.

The first proof of concept will be focused on application persistence. The team will build a storage service with GraphQL interface and demo application running on the platform which will use this service to store, retrieve and update its state. Work on building this proof of concept is underway.

The following proof of concept will be focused on external communication, which will allow applications to communicate with external servers through REST API-s, web-socket connections, and communications with smart contracts running on blockchains.

WALLET BACKEND

The work on rewriting the Cardano wallet to use the new data layer continued. The team began working on implementation of the set of endpoints for managing wallet accounts. Wallet restoration code has been refactored to use the new data layer. Also, the team concentrated on writing tests for block metadata. Implementation of the state needed to build queries for history wallet tracking continued. Furthermore, the developers completed work on static transaction size estimation needed for calculation of transaction fees. Additionally, the team had reviewed the delegation specification and made some modification to the wallet specification accordingly.

NETWORKING

The team has continued progressing on the task of Communication Protocol Design and have now transitioned from pure design into prototyping. Additionally, a team member spent time on reviewing the Peer Discovery Implementation code in order to get a better scope of the code base. After their analysis, the team member plans to create a standalone manual test where nodes will talk to each other locally.

DEVOPS

DevOps continued preparing for upcoming testnet launches. Build, deployment, and maintenance infrastructure for both IELE and Cardano are incrementally improving with each deployment and focus is shifting more towards testing. An IELE redeployment included a fix for an internal compiler error. Cardano's faucet now includes a return address.

A new release candidate for Daedalus 0.11.0 and Cardano 1.3.0 was prepared and announced internally for QA. A DevOps engineer has also been considering ways to improve the packaging and user experience for Daedalus Linux builds. A temporary misconfiguration was discovered and resolved in systems that handle Daedalus bug submissions.

Alerts indicated some Cardano mainnet and staging nodes in IOHK-controlled machines were no longer receiving blocks. Two mainnet nodes and three internal staging nodes were unreachable, and the incident was eventually correlated with a temporary networking connectivity issue that affected one of the cloud provider's regions. Both clusters restored to good health after the incident.

The team made progress on continuous integration optimizations. A proof of concept cross-compiled Cardano build for Windows was made on Linux, and an engineer verified it can sync blocks. Engineers improved build reproducible by implementing a nix derivation for NSIS tooling for Windows installers and working on a BuildKite job to validate the release.nix Hydra jobset definition. There's focus on reducing the gap between the cross compilation feature branch and develop, including discussing with developers how dependencies for all platforms can be updated while maintaining compatibility. More generally, DevOps are working on improving visibility on CI bottlenecks and coordinating with developers streamline builds.

CARDANO DECENTRALIZATION

Research and Design

The research team has completed the Delegation paper. They are now looking to submit the work to respective conferences for peer review. The design document is also nearing completion. The researchers have been specifically focused on how rewards would be collected. After the document goes through an internal review, the team plans to begin prototyping. Additionally, the specification with new delegation are nearing completion as well. The team is planning to have the first draft completed by the end of the month. Detailed Technical Implementation Plan - Several senior researchers have began documenting the implementation plan last week and continue to do so this week.

Development

The team continued refactoring the existing code base and flashing out the dependency graph. A team member concentrated on reuniting orphan instances with their types. Another developer spent time working on removing partial field constructors (and adding some JSON tests on the way). Finally, a database specialist has been rearranging the dependency graph from a db point of view.

GOGUEN

Languages and Virtual Machines

The team is now in preparations and planning for Marlowe and Plutus development.

End Users and External Developers

The Goguen team is now filling the roles of community managers, technical support engineers as well as other key roles, with the aim to provide multi-faceted support for the release.

Testnets

The week was spent preparing for IELE launch with Devops, QA, Technical Support, Marketing, Comms & Development team as well as Runtime Verification. The team reports that the IELE technology is ready for the launch and production environment.

MISCELLANEOUS WORKSTREAMS

Hardware Wallets

The team concentrated on running integration tests and reports that all of the major hardware wallet operations have now successfully passed all of those tests.

ANNOUNCEMENTS

IOHK is currently looking for talented people to work with us as a Front End developer, Exchange Experience Engineer/Coordinator, Software Test Engineer, Senior Events Manager, Testnet Community Manager, Development Experience Manager as well as several others. Please see the IOHK Careers page for more details.

July 19, 2018

Weekly Development Report

DAEDALUS

Wallet

Last week the team finished the integration of the latest React-Polymorph version in Daedalus.

In the scope of QA process for the upcoming Daedalus 0.11.0 with Cardano 1.3.0 release, the QA team discovered an issue where the "Daedalus website" and "License" links were opening within the application window instead of in an external browser. The issue was successfully resolved.

The team is working on a fix for a problem with the Electron GPU-accelerated rendering which fails on certain platforms and GPU cards/drivers combination and compatibility issues. This results in Daedalus window rendering a completely white or black screen.

React-Polymorph library was extended by a couple of new UI components: "Loading spinner" and "Progress bar indicator".

The team is continuing with the implementation of the Cardano V1 API.

App Platform

The team prepared a demo and application updated architecture diagrams for the bi-weekly stakeholders meeting. The team also completed the first iteration of the benchmarking test suite and undertook some refactoring of the application platform to make compatible with stress testing. The suite benchmarks the platform from startup to the UI rendering on screen and has been designed to evolve as the platform and bundled apps become more sophisticated. The report returns host information, total execution time, total CPU usage, total memory usage, and per-process memory usage. With 10 very minimal apps installed on a 2.3Ghz i7, the UI renders in in 2.3 seconds and consumes 1.1GB, with a single app consuming 90MB.

WALLET BACKEND

The team continued building new wallet from the formal specification by implementing endpoints for generating addresses and transactions which are now being tested.

Additionally, the team was able to successfully complete their work on the block metadata as well as extending the metadata API with the filtering operations provided by V1 API.

The team investigated ways to do error and exception handling, and testing for support history tracking and queries.

A team member finished up the tests for the static size bounds and in the process flushed out several bugs with the initial estimates. There are now Hedgehog tests covering all of the bounds and checking that the encoded length of randomly-generated values falls into the computed bounds.

Finally, the developers concentrated on porting old wallet restoration Rocks DB support to the new wallet, adding what's necessary to support functions that rely on MonadDBRead from the new wallet code.

NETWORKING

The team did further work on the communication protocol design. Several of the key team members spend last week collaborating and documenting on the protocol. A senior engineer has come up with a proposal for a protocol which is stateful server side, but which simplifies the handling of forks and their delivery to consumers. A sketch of an implementation has also been provided.

DEVOPS

DevOps continued preparing for upcoming testnets. Flaws were discovered in the Cardano testnet genesis configuration. After fixing them DevOps sought review from developers. Resource provisioning, monitoring, and production environment encapsulation are continually improving for both testnets.

Last week the team also contributed to release preparation work for Cardano and Daedalus. An engineer implemented a fix to the Linux Daedalus builds so the title bar text included the build number, as is done on Windows and macOS builds. The internal staging cluster was updated to the tip of the release/1.3.0 branch and a mechanism was added to announce on Slack the cardano-sl revision on the staging cluster whenever it changes. Around the time of this update, engineers discovered Daedalus wasn't syncing blocks from the staging cluster. DevOps investigated and determined there was likely a bug in a new block streaming optimization. The team coordinated with a network engineer to get patches and deployed to the staging cluster to verify the fixes worked. Given the change in load profile on the public relays, the team also tested many concurrent connections to the same staging relay from wallets which had to sync all the blocks from scratch. The connection count varied the relay node balanced block throughput reasonably well across all the wallets.

Engineers made general improvements to builds and testing as well. BuildKite hooks were added to ensure consistent environment settings and to clean up stale data from past builds. The nix evaluation time and memory footprint for cardano-sl, especially for mainnet NixOps deployments, was improved by using the memoise branch of nix. Several GitHub status checks for cardano-sl on Hydra were marked as required for merges to the develop branch, and internal documentation was updated to describe how the status checks are currently maintained.

On the side, the DevOps team also supported engineers with their projects. The team shared insights with a developer about how DevOps' existing wallet integration test works so they could improve test coverage for redemption certificates. DevOps also monitored mainnet while the developers working on the Icarus project ran a small test to ensure everything went according to plan.

CARDANO DECENTRALIZATION

Research and Design

Discussions were held last week on the review of the incentives paper and figuring out how to optimally flush out the content.

GOGUEN

End Users and External Developers

Last week, the product manager had started work on developing a plan to deliver features and roadmap for products in Goguen with a particular reference to end-user and external developer needs.

Testnets

The team continued with website development for the IELE testnet.

MISCELLANEOUS WORKSTREAMS

Hardware Wallets

The team finished the HD-wallet document (docs/hd.md), which can be found on the Cardano GitHub. Additionally, the developers worked on improving wallet integration tests.

ANNOUNCEMENTS

July 12, 2018

Weekly Development Report

DAEDALUS

Wallet

Last week the team finished the implementation of the improvements of Daedalus log files packing logic. Users will now always download/submit the latest application logs. The team is almost done with the integration of the latest React-Polymorph version in Daedalus as the code is in the final review and testing phase.

In the scope of maintenance tasks, the team has improved the "Escape" key handling within the recovery phrase autocomplete component in the "Restore" wallet dialog. New logic prevents the dialog from closing in case user presses the "Escape" key while recovery phrase autocomplete component is focused as the expected behavior is the closure of the autocomplete suggestions list.

The team is starting with the implementation of the Cardano V1 API. This work will involve a lot of architectural changes and refactorings and will remain in the focus of the team for the next few sprints.

App Platform

Last week the team continued the work on the implementation of the apps screens.

The team completed the basic "Terms of Service Agreement" screen, establishing a profile node in the state tree to encapsulate the logic where the platform enforces acceptance before allowing the user to access other screens.

The next screen the team is working on is the Cardano transaction demo screen. In the scope of the work for this screen the team had to configure routing, with the history persisted in the state tree. In preparation for building the minimal wallet, the team is working on the development of a service API to enable transfer of funds from one account to another. This work will continue throughout the following week.

WALLET BACKEND

API rewrite

In order to improve the wallet's error handling, last week the team started to work on the inventory of all user-facing errors in the wallet. The team is currently engaged in detailed analysis of the data. In the meantime, all HTTP/2.0 requests are blocked in order to prevent the client from triggering misleading behavior. The Target release is 1.4.0.

Wallet

The team engaged in the task of implementing new transaction endpoints. At first, the team was experiencing some CI issues which were eventually resolved and the developers are hoping to complete the task by the end of the week.

A member of the team spent ample time working on key derivation schemes and was able to construct a small model of Hardware wallets as well as provide a review of the Icarus key derivation specification.

Additionally, the team continued working on the wallet restoration from seed functionality and concentrated on porting over the parts of the code, utilizing “RocksDB” that concerns the wallet restoration.

Last week the team also investigated ways to extend queries on “Beam”, with filtering, sorting, and pagination, as well as solving some issues concerning V1 types and Kernel types.

NETWORKING

The team worked on communication protocol design. Specifically, the team spent time working on the chain selection solution and is continuing the work on an executable specification.

DEVOPS

Last week the team worked on several build efficiency improvements. Their investigation into excessive memory consumption during nixops evaluations for mainnet continues. BuildKite's macOS builds were a bottleneck, so the team reduced the scope of those agents to only on what's necessary to build macOS Daedalus installers. Nix builds were too sensitive to modifications in files which do not affect the build outputs, so they improved the filters and in turn reduced the frequency of unnecessary cache misses. Since the stylish-haskell checks are pure, the job was migrated from BuildKite to Hydra. By treating Hydra as a strictly pure build environment and with the aim to have it manage more work, the team can reduce inefficiencies and competition between it and BuildKite for build resources. To ensure failing, Hydra jobs are resolved soon after they occur, and the team worked to improve the integration with GitHub's status checks. Specifically, Hydra was patched to exclude the branch name from the status check identifiers for Hydra jobs, and this simplified how we manage branch protection in cardano-sl.

DevOps also fixed bugs, like a locale problem that affected a script on macOS and a regression in how intermediary hashes are rendered, and therefore parsed during Daedalus’ semi-automated update proposals.

Work is ongoing to support KEVM Testnet and improve how it and IELE Testnet are provisioned and maintained. We also prepared and internally announced Cardano 1.3.0 RC3's availability for testing by QA, and conducted a test update proposal on the internal staging cluster.

CARDANO DECENTRALIZATION

Research and Design

With regards to the work on Incentives, last week a senior researcher updated a section of the research paper with the current version of the "desirability formula" and the newest experimental results while another researcher looked into the "large stakeholders" question.

Development

The developers concentrated on core refactoring. While working on removing partial field accessors, the team experienced issues that require writing golden and round-trip tests for some of the data types. The developers are currently focusing on writing the necessary tests.

A specially assigned developer has almost finished relocating orphan Bi instances. When the current task is completed, they will then focus on removing unused dependencies from cabal files.

Lastly, the team is continuing to write a document proposing a fix to improve Continuous Integration build times.

GOGUEN

Fundamental Research

The team continued development for Plutus and Marlowe. They have been focusing on expanding their pool of specialists in order to push forward the development of the Plutus VM. The work on Marlowe has also progressed nicely. The team is currently working on more precise planning to optimise this effort.

Testnets

Last week was dedicated by a large part to development, infrastructure configuration and testing for IELE testnet. Aside from this, the team worked on website and content creation for IELE testnet. The team is currently bringing on-board a technical support engineer.

MISCELLANEOUS WORKSTREAMS

Hardware Wallets

Last week the team dedicated a lot of their time to creating various documentation as well as continuing unit and integration testing.

Additionally, the team has been working closely with a blockchain consulting firm to achieve compatibility with the new BIP-44 (Bitcoin Improvement Proposal) style which is planned for implementation in future versions of the wallet as well as the upcoming light client.

ANNOUNCEMENTS

July 5, 2018

Weekly Development Report

DAEDALUS

Last week the team finished all "Testnet" related tasks. It is now possible to install network specific Daedalus builds (for example, the testnet and mainnet) and run them in parallel on the same machine. These features will be delivered as a part of Daedalus 0.11.1 with Cardano 1.3.1 release.

The team has improved compatibility of the wallet address QR-code printed on the paper wallet certificate. Improved QR-code should now be working with all existing QR-code readers.

In the scope of maintenance tasks, the team has refactored wallet deletion code to prevent race conditions where Daedalus would try to fetch fresh data for the wallet which is in the process of being deleted. The team is working on improvements of Daedalus log files packing logic with the goal of making sure users always download/submit latest application logs.

Last week the team released React-Polymorph 0.7.0 which introduces a lot of architectural improvements, bug fixes, and new features. The team is currently finishing the integration of this latest React-Polymorph version in Daedalus.

WALLET BACKEND

The team continued the efforts to build new wallet backend from the formal specification.

Work on transaction submission layer, which is in charge of submitting and scheduling transactions to the blockchain, was completed and merged.

Automated tests were written for input selection algorithm which was completed last week.

Work on new build system based on Cabal continued. Stack2cabal tool for translating Stack configuration to Cabal was written. New build system works well on Linux but there are linker issues on macOS which need to be resolved.

Additionally, work on wallet restoration from seed and block metadata continued.

NETWORKING

The networking team continued their efforts to speed up block verification.

DEVOPS

The DevOps team made improvements to build infrastructure last week. Memory usage has been high on nixops mainnet deployments, so we added diagnostics and investigated potential improvements. Nix build caches for cardano-sl were getting invalidated too much because the current git revision is used by the executables. By injecting the git revision into the output artifacts from the bulk of the build, spurious cache invalidation is avoided. We also made Plutus buildable using nix and integrated it into Hydra for continuous integration.

Preparations for additional testnets are underway, with a focus on the IELE testnet and the Cardano testnet which reflects the code currently running on mainnet. There have been internal deployments for both, and we are iteratively validating the configurations, improving the code, updating the environments. Support for running an independent copy of Daedalus for Cardano testnet is improving and getting more polished.

CARDANO DECENTRALIZATION

Research and Design

After addressing the previous week's testing issues, the Incentives team feels confident with the results of the latest tests performed. The team has concluded that the biggest missing item for the paper is now the "large stakeholder" question. The questions is what they will/should do. Split their stake into several pools? Run several pools? The team is now engaged in analyzing this and writing a section in the paper about the results, as well as updating the chapter to outline the results of the experiments. They have changed quite a lot since it was written, so it needs to be updated with the newest formulas and results.

Development

Several of the developers have teamed up and successfully finished off the golden/roundtrip tests for serialization and deserialisation.

While that task was being addressed, another member of the team has finished replacing deriveSimpleBi variables with deriveIndexedBi variables for sum types.

Additionally, the team worked on replacing type constraints with explicit parameter passing.

GOGUEN

Fundamental Research

The research pertaining to Marlowe, Plutus as well as Sidechains is reported to be progressing nicely.

End Users and External Developers

The newly hired product manager has continued familiarizing themselves with end user and external dev requirements, by researching feedback from the community as well as reported issues.

MISCELLANEOUS WORKSTREAMS

Hardware Wallets

The Ledger backend APIs have been built against the existing wallet backend. Unit and integration testing are ongoing.

Daedalus App Platform

Last week the team finished all quality assurance tasks.

The team discovered a couple of rendering problems regarding the way text fields are managed under RJX. The problems were resolved by replacing RJX with a custom renderer which makes sure only changed components are updated and re-rendered.

The team implemented a client-side state-tree as the foundation for building out the UI features, in a format that is suited for saving and recovering during a restart. This change also cleans up the remaining hard references used for prototyping.

The team has started with the implementation of the apps screens. First of which is the one where the user needs to accept terms of use before using the software. This task will be in the main focus of the team for the upcoming week.

ORGANIZATION

The software development management team spent last week on creating new organizational workflows. Haskell Engineers are now organized in small, autonomous, self-organizing teams called Squads.

Each Squad is lead by a Squad Lead and has full responsibility for delivering their respectively assigned features (including specifications, coding, testing). The development process is a queue based process similar to Kanban and is fully integrated with the Project Management Processes.

The team developed tools for daily progress monitoring and is now fully employing these tools. All development activities are now tracked in a customized Issue Tracking System. Cycle Time (duration of time from start to end development of a feature), Flow Efficiency (amount of time a feature is worked on vs. not worked on) and Cumulative Flow Diagrams (An area graph that depicts the quantity of work in a given state, showing arrivals, time in queue, quantity in queue, and departure) are measured on a weekly basis.

ANNOUNCEMENTS