Skip to content

[#666] Add Quality Declaration #667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
elBoberido wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[#666] Add Quality Declaration #667

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions QUALITY_LEVELS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Quality levels

The quality levels for the components in iceoryx2 are derived from the
[ROS quality levels](https://github.com/ros-infrastructure/rep/blob/master/rep-2004.rst).
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand the contents of the provided links: no they are not, they are inspired by them but I would not link this here so explicitly.

Despite developing some targets according to automotive standards like ISO26262,
the code base standalone does NOT legitimize the usage in a safety-critical
system. All requirements of a lower quality level are included in higher quality
levels e.g. quality level 4 is included in quality level 3.

## Quality level 5

This quality level is the default quality level. It is meant for examples and
helper tools.

* Derived from [ROS quality level 5](https://www.ros.org/reps/rep-2004.html#quality-level-5)
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would get inspired by those quality levels but not link them here directly since this does not make sense for us. Maybe it makes sense to say that quality level X satisfies all requirements from ROS2 quality level Y but as I see it we exceed even with examples quality level 3 of ROS2.

* Reviewed by at least one approver
* No compiler warnings
* License and copyright statements available
* No version policy required
* No unit tests required
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add here: Usage documentation, no code documentation.


## Quality level 4

* Derived from [ROS quality level 4](https://www.ros.org/reps/rep-2004.html#quality-level-4)
* Basic unit tests are required
* Builds and runs on Windows, macOS, Linux
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would state that it runs on every supported platform and then we link to the list of supported platforms.


## Quality level 3

* Derived from [ROS quality level 3](https://www.ros.org/reps/rep-2004.html#quality-level-3)
* Documentation required
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I follow the link, then it states:

  • Must have a declared license or set of licenses
  • Must state copyrights within the project and attribute all authors
  • May have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements

All of this is already done at quality level 5.

* Test specification required
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope: See link, No explicit testing requirements, though covering some if not all of the features with tests is recommended

Also what is a test specification?

* Version policy required

## Quality level 2

This quality level is meant for all targets that need tier 1 support in ROS 2.
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove this statement and would in every quality level maybe add: conforms with ros2 quality level X.

Also we could state then things like: conforms with ISO 26262 ASIL-D (later) and add also other standards.
Those things have an actual meaning.


* Derived from [ROS quality level 2](https://www.ros.org/reps/rep-2004.html#quality-level-2)
* Must have a [quality declaration document](https://www.ros.org/reps/rep-2004.html#quality-declaration-template)

## Quality level 1

* Derived from [ROS quality level 1](https://www.ros.org/reps/rep-2004.html#quality-level-1)
* Version policy for stable API and ABI required
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also link our version policy (semver) so that it is clear what is meant by that.

* [ASPICE](https://beza1e1.tuxen.de/aspice.html) SWE.6 tests available
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided link does not state anything about SWE.6 tests, what are they, how are they defined. This needs to be at least linked.

* Performance tests and regression policy required
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a regression policy.

* Static code analysis warnings addressed
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quality level 5 but we can restrict this to: Rust (clippy, no compiler warnings and miri), C++/C (clang-tidy, address/undefined behavior/thread sanitizer, no compiler warnings)

* Enforcing the code style is required
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quality level 5. This is a minimum.

For this and the static code analysis we should define what we mean by that. In this case, follow clang-format or rustfmt.

* Unit tests have full statement and branch coverage
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't full statement coverage a subset of branch coverage? So branch coverage would suffice?!


## Quality level 1+

This quality level goes beyond the ROS quality levels and contains extensions.
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this line. I would not reference ROS2 here. They are, in my opinion, not a valid reference for code quality.


* Code coverage according to
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add here:

  • Requirements
    • every test is linked to a requirement bidirectional
  • detailed design documents
  • detailed use cases

[MC/DC](https://en.wikipedia.org/wiki/Modified_condition/decision_coverage) available
22 changes: 22 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
* [listener.rs (grabbing all events at once)](#listenerrs-grabbing-all-events-at-once)
* [Custom Configuration](#custom-configuration)
* [Supported Platforms](#supported-platforms)
* [Quality Levels](#quality-levels)
* [Language Bindings](#language-bindings)
* [Commercial Support](#commercial-support)
* [Thanks To All Contributors](#thanks-to-all-contributors)
Expand Down Expand Up @@ -321,6 +322,27 @@ The support levels can be adjusted when required.
* **tier 2** - Works with a restricted security and safety feature set.
* **tier 3** - Work in progress. Might compile and run or not.

## Quality Levels

The repository contains multiple crates, tools, examples, and benchmarks. The
quality of these components depends on their intended use in a production
environment and on the target platform. Below is an overview of the current
quality levels, as well as the target quality levels for tier 1 and tier 2
platforms.

[Quality level](./QUALITY_LEVELS.md) are 5 to 1+, where 1+ is the highest level.

| Component | Current Level | Target Level for Tier 1 Platforms | Target Level for Tier 2 Platforms |
|-----------------------|:-------------:|:---------------------------------:|:---------------------------------:|
| iceoryx2 | 2 | 1+ | 1 |
Copy link
Contributor

@elfenpiff elfenpiff Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use nowhere quality level 3. Should we then merge quality level 2 with 3? Also 1 and 1+ are a bit weird.
I would make this its own normal quality level.

So it would be:

  • 1+ -> 1
  • 1 -> 2
  • 2 and 3 -> 3
  • 4 -> 4
  • 5 -> 5

| iceoryx2-bb | 2 | 1+ | 1 |
| iceoryx2-cal | 2 | 1+ | 1 |
| iceoryx2-cli | 5 | 4 | 4 |
| iceoryx2-ffi | 2 | 1+ | 1 |
| iceoryx2-pal | 2 | 1+ | 1 |
| benchmarks | 5 | 5 | 5 |
| examples | 5 | 5 | 5 |

## Language Bindings

| Language | State |
Expand Down