QA of charms in charmstore?


#1

Is there presently some quality assuring process in charmstore in place or discussed? @rick_h


#2

No, there’s not a global promise around the charms. It depends on the charm and the community around the charm to handle their development and QA practices as they build charms and go through the edge to stable channels.

In a way it’s like github or other hosting sites. In the end you end up learning who you trust I guess.


#3

Right. I fear that a complete lack of QA places alot of charms in the low end and affect the “general feel” of juju itself.

I think that it would make some sense to have perhaps a “optional” test-suite automated that indicates if a charm passes some minimal QA checks.

I envision, for example, charms getting “rated” according to some badge system. That would encourages charmers to add an extra touch of quality without for that sake enforcing a QA.

The most pro charmers will end up having nice looking “badges” for their charms in the store based on the quality score or tests they reach/pass.


#4

One thing that Github provides–as flawed as it is–is a count of stars/forks. That tends to provide some signal about the package’s quality. It’s also pretty easy to detect how alive the project is because the code is so close to the UI. Additionally, many repositories embed stickers from CI tools into their README.

It would certainly be nice to emulate some of that functionality. At the moment, it’s very difficult to detect what level of quality there is in each of the charms.


#5

This is definitely something we can look at as we look at ways to build a common experience from the snap store through the Juju Charmstore.


#6

I agree with all this. When I look at a Github page, I look at general activity as well as frequency of updates, how recently it was updated, stars, forks, watchers, issues, commits, freshness of PRs, then the quality of the readme, with specific focus on build/install/deploy/integrate instructions. It’s a reasonable initial gut check. Agreed it’s flawed, but there’s enough metrics for me to get a good feeling for what I’m about to bite off.

On the charm store it’s really hard to see how recent something was updated (almost all outside of the promoted charms that I look at are very stale and broken, but that takes considerable effort to discover), forking a charm to upgrade series or something else simple is often challenged by unresolvable interface deps. Generally any charm with a series <Xenial is simply broken and unusable, if it appears in the ‘recommended’ section, in my experience (admittedly my window of use case is very blinkered, but some stats revealing the contrary would be welcome). In the community section a series-upgrade or charm pull and some simple tweaks often will suffice so long as the charm complexity is low.