Status Updates

Weekly Development Report

CORE

• Exchange Onboarding & Support

For smoother communication with exchange, the team has created a dedicated internal channel, that will be used for communication between the development teams. In the past week, the team has moved forward to finalise V1 API and test coverage for the initial rollout . The task of adding a V1 endpoint to retrieve address meta-information (CSL-2327) is mostly complete and is currently pending some API design decisions. Feedback about V1 exchange endpoints from front-end team (CSL-2335) was successfully collected, and almost all of the requested changes are pending review. The team concluded that the task of servant-quickcheck for wallet-new API (CSL-2338) was not resource viable and the development resources would be better allocated towards other work. The task of making sure that cardano-node-new can log the same information of the original cardano-node (CSL-2347) was completed and reviewed, but has not been merged yet. Meanwhile the effort to write logical integration tests for wallet key import endpoint (CSL-2371) was completed, though actually wiring up the tests with the finished client is pending (CSL-2398). The team reports that the task regarding wallet API which needs to be reviewed after Ntp refactoring (CSL-2375) is ready to implement. The team has decided on a design and are waiting on the CSL-2346 pull request to be merged. The work to collect missing V1 endpoint data compared to V0 endpoints (CSL-2400) has been completed, with an impressive and comprehensive report. CSL-2335 implemented some of the noticed differences, but further work stories can be generated to implement the remaining tasks. Some of the endpoints haven't been implemented in V1, notable /v1/addresses. The final push to finish this work is tracked in CSL-2416 and will be complete next sprint.

DAEDALUS

Due to the changes in paper wallet certificate generator specification the team had to revise and modify the original implementation. The feature is getting polished and prepared for external audit. The team has also been working on preparations for the new release of React-Polymorph which includes a set of new components and a full structure refactoring.

WALLET BACKEND

• Work on current wallet backend

One of the areas the team concentrated on last week was the effort to implement wallet backend's new data layer following its formal spec. A team member is nearly finished with CSL-2382, CSL-2378, and CSL-2395. Another team member has already reviewed a lot of this work, and worked closely with the original specialist team member in the original design. A pull request is not yet ready, this is pending CSL-2382. However, the team reports that good progress has been made in regard to that pull request. Furthermore the team reports that they have made good progress on CSL-2381 Support restoration at the Wallet Storage level. The planning for this ticket itself is has improved significantly. The developers have went through the entire V1 REST API specification, extended the outline of this ticket with the stuff that was missing, and tried to come up with time estimates for all the subtasks, which were discussed with a senior engineer and a senior software manager on this task. Additionally the task of adding server side TLS check has been successfully completed (see CSL-2683). Development / Testing certificates in scripts/tls-files have been updated, and a client certificate has been added for testing. The servers-side checks also include a client's name verification (information embedded in the client's certificate, matching against values provided to cardano-node via the CLI, default to "Daedalus Wallet"). Certificates are generated from a script located on the Daedalus repository. This script has been slightly adjusted and a PR will be issued soon to cover the changes. An end-user documentation extension has been submitted to describe the required manipulation of generating and using new certificates. Also, the team progressed in the effort for asynchronous restoration of a wallet from seed. Time has been spent working on CSL-2246 ( Allow correct pending transaction handling on a restored wallet). It didn't go as well as planned and therefore the task was not complete due to some complication in the current design. What was completed was to pivot the scope of the task a bit, and opened a work-in-progress on pull release which is an effort to support restoration at the Wallet Storage (CLS-2408) Once merged, this PR will allow the team to complete the task of extending the Wallet Info state to report the restoration progress (CSL-2247) in such a way that frontend will have enough information to:

--- Show the user that a wallet is restoring, and which % of progress it reached;

--- Grey out some parts of the UI to prevent the restored wallet to send transactions.

The team feels that is is very crucial to provide this extra information as part of the V1 API.

NETWORKING

The work on cleaning up/improving the NTP subsystem is coming along. The effort is almost complete. However, the work needs to be run by and pass the quality assurance guidelines set by the team leaders.
Furthermore, the team is engaged in the task of block streaming. As of yet the team has not been able to get a test/simulation up and running due to the fact that the team leader decided to increase emphasis on work quality, rather than speed. This means doing a fair amount of remedial work that makes the test/simulation setup easier and less awkward. It also greatly simplifies the code base. A team specialist was able to review some of the work, which necessarily touches more than just networking, as the changes were foundational, correcting some mistakes which were deeply ingrained.

DEVOPS

The team has addressed some of the issues from audit report, automated installer generation in mind with Linux installer, dealt with some non-impactful secret leakages, continued work with V1 api for exchange documentation

MISCELLANEOUS WORKSTREAMS

• K/IELE

Work progressing; good work by Mantis team in terms of code merging, building tools for testnet and supporting Runtime Verification.

• Sidechains

Research continuing and expect to submit paper to conference in May 2018.

• Smart contract language

The smart contract language team's research work continues and interviewing of candidates progressing well.

• Marlowe

Marlowe research paper draft completed and being reviewed internally.

• Misc.

Asset locality paper nearing completion; workshop on design in April 2018. Accounting model work completed in terms of academic work.

Tech Support

Last week saw a reduction in the number of tickets received. The tech support team received a total of 457 tickets, out of which 326 were successfully resolved and the remaining 131 are being followed up. The top reported issues were:

• Unable to connect to network
• Unable to update wallet
• Cannot sync
• Misc.

Weekly Development Report

CORE

This week the team concentrated on merging remedial work from the master to the develop branch. This work was split into two pull requests. The first part of the PR which is to include the DB and mempool consistency work, which is complete and was successfully merged. The second part of the PR is to port DB type conversions and speedups. It has been analysed and documented and is currently under review. Additionally, several members of the team worked on panning out required tasks for creating a log file analyser for bug classification. The work has been assigned to a resource and needs to be planned in for this week. Furthermore, the effort continued toward an upgrade to GHC (Glasgow Haskell Compiler) 8.2.2. Currently, the last task is to merge the PR. However, this is currently blocked by 2 CI errors. The team plans to resolve these errors in the coming week and complete the upgrade. Also, the team pushed to improve block syncing. Progress made on improved block syncing which will be to stream as opposed to batch the blocks. The team hopes that this will give them better performance overall. Initial implementation is complete and currently working on test setup. Plan for this week is to review and tune the code.

• Exchange Onboarding & Support

One of the past week’s main goals was to release a wallet to exchanges, which exposes (a subset of) the V1 API. The team reports that good progress has been made towards this effort. Implementation of the API client testing was successfully completed.The team is currently proceeding with Integration tests. Importing of an initial wallet key is in progress. As for filtering by transaction timestamp and ID, a majority of this task is completed and the rest is planned to be completed this week.

DAEDALUS

Implementation of the paper wallet certificate generator is complete. The team is working on the paper wallet certificate restoration feature which is required in order to restore a wallet from a paper certificate. As a part of this task, the the React-Polymorph library autocomplete component had to be extended to support clear active selection feature. As a part of regular maintenance, the team has also updated third party dependencies such as moment.js to the latest available version.

WALLET BACKEND

The team is putting in time on adding a server side TLS check. The developers are currently finishing up the last bits of the code. The server certificates hostnames were too strict, therefore the team have updated the build certificates from Daedalus to be more lenient towards server accepted names. Changes to the library were also made in order to accept alternative names, however at the present time this is hardcoded and the team will need to look for a more flexible approach. Discussion between the team will happen this week in regards to TLS authentication/integration to agree on an approach.

NETWORKING

The team is currently working on block syncing and speeding up verification and application of blocks, which is planned to complete in the current sprint.

DEVOPS

The team is engaged in testing the Linux installer internally, so far with no known issues. The team successfully leaned up some technical debt and is heavily working on exchanges V1 documentation.

MISCELLANEOUS WORKSTREAMS

• Sidechains

Sidechains work continues with simulation results provided as input to research to support a paper for submission at the start of May. Early design work and planning on implementation of the sidechains team to start shortly.

• Smart contract language

Continuing with fundamental research necessary to define the boundaries/scope of priorities for developing the language. Key milestones in the research phase established and detailed planning is scheduled to start in due course. In hiring process a good list of potential candidates has been assembled for the team.

• Reagan

Reagan research continues with over 43000 tests run against KCC; report to be updated next week and review findings by end of month.

Tech Support

Last week saw a reduction in the number of tickets received. The tech support team received a total of 611 tickets, out of which 263 were successfully resolved and the remaining 348 are being followed up. The top reported issues were:

• Unable to connect to network
• Unable to update wallet
• Cannot sync
• Misc.

Weekly Development Report

CORE

Last week, the team prepared a pull request, aimed to finish the upgrade to GHC 8.2.2 (Glasgow Haskell Compiler). However, the pull request is currently blocked due to two CI errors. The team has requested help from DevOps, and is currently looking for a solution. A proposal pertaining to creating specification for efficient block storage has been created and is awaiting review. The task of fixing the block version update has been delayed and is not planned to be completed during the current sprint. Additionally, benchmarking has been successfully completed for the HTTP-based block sync proof-of-concept. As for investigating and fixing issues pertaining to reliable shutdown, a new team member has been onboarded and is now contributing to the effort. Also, the segfault in rocksdb code which occurred during a shutdown is now corrected. Previously, stopping cardano-node using CTRL-C used to take one min. This has been investigated, a possible fix has been found and help has been requested from a specialist team member. The team feels confident that this problem will be fixed within one or two days.

• Exchange Onboarding & Support

The team also tackled the task of documenting the new Wallet V1 API. The subset of V0 API ported to V1 has now been fully documented. This documentation includes a "Walk-through Wallet Manual" explaining basic use cases. The documentation is served via a dedicated endpoint on the API. New use cases are to be added in the coming days. A final review of the documents (typos, wording) will be done early next week. Finally, all needed endpoints for exchanges are ready to be ported to V1. Currently, the code can be used by the exchanges as a pre-beta version (this is done at their own risk). Testing has begun but is not yet complete. The team plans to gather feedback on the API (code and doc) from the exchanges in the coming days.

DAEDALUS

Implementation of the paper wallet certificate generator is nearly done. The team is working on code review and testing of this new feature.

WALLET BACKEND

• Work on current wallet backend

Last week the team invested a considerable amount of time on decommissioning the Cardano-SL-1.0- exchanges branch. Unfortunately, the DevOps team did not have time to work on this user story due to higher priority tasks. The Wallet Backend Benchmarking code has been packaged as a viable solution and can be used as a standalone benchmarking tool by everyone who needs it. Currently, that tool can be used to test API V0 only. It has been designed in such a way that it can (and will) be extended to work with API V1. Optimization work has a PR which still needs to be closed and then this user story can be considered as done. Also, the team engaged in the effort of adding server side TLS checks. The code has been written, however, currently the performance is not satisfactory. The team feels that they need to assign a specialist with intimate expertise in TLS. This task has been put on hold, due to the fact that the needed specialist is currently assigned to a higher priority task. A prototype, capable of not only restoring the wallet balance but also the wallet history incrementally has been developed in order to achieve asynchronous restoration of a wallet from seed. This prototype is currently being tested. After the tests have been successfully finished, the team feels that the following tasks will remain to be done:

-- Expanding of the (V0|V1) endpoints to reflect the restoration status.

-- Making sure all the edge cases are properly handled.

-- Writing of the CT (Cardano Test project) tasks for smoke testing/ integration testing.

• API rewrite

The pull request aimed at design and implementation of the /api/v1/transactions endpoints is still open and is awaiting review.

NETWORKING

The team is currently working on Block Syncing and NTP SUBSYSTEM (ISSUE WITH the 1.1.0 release)

DEVOPS

The team successfully created a pull request with a working Linux installer. Next up is a code review and internal testing. Last week’s 1.1.0 release deployment took two days and went mostly well. The team managed to migrate the remainder of the legacy infrastructure and thus successfully completed a major chunk of work. The team also provided support pertaining to Zendesk classification automation and benchmarking infrastructure.

MISCELLANEOUS WORKSTREAMS

• Delegation

An initial pass of the research paper has been completed and reviewed. A request has been made to have an internal paper produced on Ouroboros rules and lightweight certificates. Currently, this request is awaiting approval from senior management and a subsequent assignment of necessary resources. A decision has been made by senior management that exchange addresses are required to be included in the delegation scheme. However, the team still need to meet to discuss the requirements.

• K/IELE

The team continues to make good progress with the Mantis team, evolving design on how the testnet will work, modularising code for better maintainability, supporting QA, building tools for RV to test IELE and creating components needed for the testnet. The RV team is making reasonable progress but the lack of precise details on testing status and errors is making it difficult to know exactly how ready their components are. Differing approaches in terms of documenting issues, software QA and project management are causing some challenges, however these will be resolved over time. The project manager needs to provide teams with clarity on testnets (RV and IOHK).

• Sidechains

The Sidechains team had an important meeting with engineering and there is an evolving plan on how to get the experimental results needed for the May paper submission.

• Smart contract language

The team have started using YouTrack to keep track of ideas and research tasks they are performing. The intention is to set a target for the design proposal to be presented to senior management when the team is next available (team members are travelling or attending conferences this week).

• Other developments

Reagan has also had some good progress in terms of test cases executed (over 6000 more) and understanding the meaning of some (now determined) spurious results. Additional compute capacity is needed for the next stage, provided that it will get approval from the stakeholders. The Marlowe team have revised the conference target. They decided not to submit to a crypto conference, however, the paper is planned to be completed in March. Presentations are planned for early April on progress, a workable demo and a plan. Asset locality and Accounting has seen completion of the extended work on UTxO formalisation to cover Bitcoin-style redeemer/validator style addresses and transactions. Work has begun on asset locality design.

Tech Support

Last week saw an uptick in the number of tickets received. The tech support team received a total of 1625 tickets, out of which 279 were successfully resolved and the remaining 1346 are being followed up. The top reported issues were:

• Unable to connect to network
• Unable to update wallet
• Antivirus-related issues
• Cannot launch
• Transaction-related questions

Weekly Development Report

CORE

Last week, one of the key points of the team's effort was the upgrade to Glasgow Haskell Compiler 8.2.2. The team concentrated on performance regression on the build with GHC-8.2.2, to make sure that the build time for Appveyor CI was increased to 90 minutes. Additionally, current conflicts with the develop branch are being solved and retried on the CI. Benchmarking for HTTP-based block sync proof-of-concept was also completed in the past week. In regard to format specification in machine-readable logging: The last two issues, outlined in the Logging Proposal document have been solved, making the document clearer. A senior engineer is due to give the document one last review.

• Exchange Onboarding & Support

In regard to documenting the new Wallet V1 API: Work has resumed and is progressing as quickly as possible. A test strategy has been defined for porting all the required exchanges’ endpoints to V1 all the endpoints. Servant-client will be used to run tests against V1 API. This has lead to the creation of more tasks related to error handling and test writing.

DAEDALUS

Implementation of paper wallet certificate generator for Cardano wallet continued and was the main focus of the week.

WALLET BACKEND

• Work on current wallet backend

The team put in a lot of effort into adding a server side TLS check. The first variant of the server side TLS check has been implemented, however it doesn't yet work as expected. A pull request, regarding DB schema change and migration to use more appropriate representations, is ready for review. Furthermore, the team addressed the task of asynchronous restoration of a wallet from seed. The asynchronous restore is implemented. The wallet shows the balance in a matter of seconds (~20 on staging). However, due to the legacy of the old wallet, there are edge cases which are still not working correctly. These will be addressed during this week. The endpoint to report to Daedalus the transaction syncing status is yet to be implemented.

NETWORKING

In regard to speeding up verification and application of blocks, the various development efforts have been merged into the develop branch. Discussion with reviewers is now ongoing.

DEVOPS

Last week we resolved an incident on staging which resulted into 16 action items to improve our process and tooling. We deployed a couple of staging updates to move us closer to a stable mainnet release. We are continuing to improve/unite our CI and are adding more automation for quicker release cycles.

MISCELLANEOUS WORKSTREAMS

• IELE and K-EVM

Good progress was made on the Mantis portion of K-EVM with the team addressing issues raised by RV, refactoring and consolidating code, adding to the test suite, fine tuning the API and assisting QA to familiarise them with the system. Progress on RV side has been mixed with little improvement on the Solidity-to-IELE compiler and unsubstantiated IELE VM progress. It seems impractical to give a countdown timer for testnets at this point. The team is currently working with the RV PM to put the QA process in place (internally) and getting the IOHK QA team in position to validate their work. The team is currently collaborating on the subject of what is the highest priority, with DevOps and others on testnets (IELE VM and K-EVM).

• Smart Contract Language

The SCL is still in an early stage. However, the team is very motivated and starting to document work they are performing as well as a plan for development.

• Other work

Marlowe is also showing good progress with the paper and a demo is expected by the end of March 2018. Sidechains has not been progressing well in the last week and more work is needed to meet May 2018 target. Reagan work is progressing with additional testing and healthy discussions between key stakeholders. Asset Locality and Accounting research is proceeding with extended work on accounting to finish this week and asset locality work to move forward. An internal document is expected by mid-March.

Tech Support

Last week saw a reduction in the number of tickets received. The tech support team received a total of 592 tickets, out of which 351 were successfully resolved and the remaining 242 are being followed up. The top reported issues were:

• Unable to connect to network
• Unable to restore wallet
• Antivirus-related issues
• Cannot launch
• Transaction-related questions

Weekly Development Report

CORE

• Core Backlog

This week the team concentrated on various key tasks. One of those tasks was the upgrade to Glasgow Haskell Compiler v8.2.2. A key member of the DevOps team noticed performance regression on the build with GHC-8.2.2. To mend the issue, another member of the team updated to the latest develop branch and made sure that the latest versions of libraries are used, including the log-warper fix by a member of the Networking team. Furthermore, the team put effort towards tackling HTTP-based block sync proof-of-concept. As it stands, block downloading on cardano-sl level has been tested. However, work on this feature will be stopped as the progress in the network layer has made it obsolete. In regard to Machine-readable logging: specify format two members of the team reviewed the logging proposal document and raised some concerns about preserving the human-readable format. At this moment there are two main issues left unresolved: Should the developers use only structured logging, human-readable logging or both as well as should the team implement structured logging in log-warper or just use another library for structured logging? The team is currently discussing the pros and cons of the respective approaches.

• Exchanges Onboarding & Support

This week a team member has been officially assigned as the first line SPOC (Single Point of Contact) for the exchanges. The new SPOC contacted an exchange and the integration team received a list of questions from the exchange, which will be addressed in the documentation. Additionally, first agreement on the various communication channels between exchanges and IOHK has been put into effect.

DAEDALUS

Implementation of paper wallet certificate generator for Cardano wallet kicked off. This feature will be delivered within the scope of the Daedalus 0.10 and the Cardano 1.2 release. Testing for the upcoming Daedalus 0.9 and Cardano 1.1 release was completed, and the release is ready to be shipped. The team participated in the Daedalus roadmap planning workshop where the longer-term roadmap was defined.

WALLET BACKEND

• Work on current wallet backend

In the past week the team worked on the task of DB schema change and migration to use more appropriate representations. A pull request has been created and has already undergone through a couple of review passes, but the test of migrations seems to be failing on Windows CI. On the front of Wallet Benchmarking, Performance Analysis and Improvement: A member of the team has started running benchmarks last week to compare the performance of the remedial work. So far, the results are not clear as the patched wallet seems to be running slower than the old one. The team plans to investigate further with profiling enabled, to see if there is anything obvious that stands out. In regard to Asynchronous restoration of a wallet from seed: the team has reached their first agreement in terms of API design. However, the actual implementation has not yet commenced and is planned to be started shortly.

• API rewrite

One of the tasks tackled this past week has been the task of Porting (and revision) of /api/test/x endpoints. Both api/test/x endpoints (CSL-2094 + CSL-2093) have been successfully implemented. Some end points have been manually tested whereas others require unit tests which are currently in progress. The pull request in regard to the task of designing & implementing /api/v1/transactions endpoints is ready for review. Last, (but certainly not least) the development and coding of functionality that would allow users to send bug reports directly from Daedalus has been completed and the final tests are being completed.

NETWORKING

The team engaged in tackling the task of Block streaming. Prerequisites have been completed and tested. Implementation of block streaming has started and is going well. Additionally, network-related bugs are being investigated and solved.

DEVOPS

The team has made a second pull request for the Linux installer, which is ready - implemented via the nix-bundle. What's missing is a proper review, refactoring and internal testing. The team is making progress by using Dhall for configuring the cardano-launcher, which would give the ability the installer builds across OS and environments (mainnet, staging). Last but not least, the team is currently in preparation for the Nix 2.0 migration.

MISCELLANEOUS WORKSTREAMS

• Sidechains

No progress since last week due to team availability. The intention is to still propose a paper for a conference in May 2018. The engineering review on the sidechain paper is still pending.

• Smart Contract Language

Recruitment for roles promoted and in progress with over 13 candidates in the pipeline. A use case for crowdfunding has been identified, intended to guide design decisions. A discussion was also had pertaining to how the smart contract language could address limitations in Solidity including testability. The team discussed two approaches to maintaining state in Cardano SL: (1) small changes to Cardano SL-style transactions OR (2) extra space for program state to transactions that on-chain code can update. A team member proposed experiments to test both approaches using the crowdfunding use case.

Tech Support

Last week saw an uptick in the number of received tickets. The tech support team received a total of 823 tickets, out of which 625 were successfully resolved and the remaining 198 are being followed up. The top reported issues were:

• Unable to connect to network
• Antivirus-related issues
• Cannot launch
• Transaction-related questions