2021 Q3 Hyperledger Caliper

Created by Attila Klenik, last modified by Angelo De Caro on Dec 09, 2021

Project Health

  • Two maintainers and one significant contributor (soon to be maintainer) work on the project.
  • An ongoing mentorship project targets a new feature that will lower the entry barrier for defining workloads.
  • Rocket.chat activity is mainly about the configuration and setup of Caliper.

Required Information

  1. Have you switched from master to main in all your repos ? Yes
  2. Have you implemented the Common Repository Structure in all your repos ? Partially

Questions/Issues for the TSC

No issues at this time.

Releases

v0.4.x patch releases.

Overall Activity in the Past Quarter

  • Identifying limitations that could hinder the implementation of complex, large-scale benchmarks.
  • HL Mentorship contributions: gradually implementing parts of a built-in, declarative workload module.

Current Plans

  • Finish, integrate, and document the HL Mentorship project. Reach out to the community for feedback.
  • Implement the common repo requirements.
  • Migrate the CI/CD pipeline from Azure to GitHub Actions.
  • Make the orchestration, communication, and round execution of worker services more flexible to allow for more complex benchmark implementations.

Maintainer Diversity

Attila Klenik: Maintainer Nick Lincoln: Maintainer Dave Kelsey: Contributor (soon to be maintainer)

Contributor Diversity

Primary activity from IBM contributions, smaller PRs being raised by the community.

Additional Information

Reviewed By

Submission date

29-Oct-2021

Comments:

Re: mentorship project, please feel free to make use of TSC meeting as well. Updating the mentorship project might bring in interest from other project maintainers.

Posted by arsulegai at Nov 01, 2021 12:35

Attila Klenik  : can you comment on what remains to be completed in the common repository structure?

Posted by tkuhrt at Nov 01, 2021 13:53

Technically, the required files are present (pending a minor link fix), but some of them need a content update (e.g., contributing, maintainers). So no major issues, just some polishing.

Posted by klenik at Nov 02, 2021 12:38

> Identifying limitations that could hinder the implementation of complex, large-scale benchmarks.

Is there a folder/list with suggested benchmarks or workloads th"at could evaluate the performance of different platforms to provide the baseline?


Posted by C0rWin at Nov 04, 2021 14:08

Artem Barger I would use the word "baseline" carefully when talking about comparing different platforms. The applied workload characteristics (its attributes and rate) can significantly influence the effective performance of a platform. So I would define a performance baseline in the scope of a specific workload using a "default" system setting (whatever th"at means). Then you can start tweaking the system settings and scale to up the performance in the context of the selected workload. There won't be a one-fits-all setup due to the diversity of possible workloads.

To answer the actual question: I'd suggest taking a look at the SmallBank workload in the Caliper benchmarks repo. The workload's attributes are relatively complex and well defined, and you can tune the workload rate through Caliper. It has a corresponding Fabric Golang chaincode implementation , and it probably wouldn't be difficult to create a Solidity-based implementation too (e.g., for HL Besu, or other private Ethereum client implementation, supported by Caliper).

Currently (but hopefully not for long, spoiler alert (smile)), SmallBank is the most representative and openly available benchmark (regarding its complexity) we have for DLTs. But if you have any information about other complex DLT benchmarks, we'd be happy to take a look at and include them in the repository in a Caliper-compatible format.

Posted by klenik at Nov 09, 2021 10:32