Smokey's Security Weblog

veritas odium parit

Matousec’s Firewall Challenge wrinkle: conflict of interests?

From Matousec’s home page:

Firewall Challenge recommendations (2008/04/20 15:16)  Firewall Challenge revealed several notable security products. We have decided to recommend the best products to you via affiliate programs of their vendors. Every product that scores at least 80% in the challenge and thus receives our mark of Very good or Excellent Protection level may become a recommended product. We contact the vendors of such products in order to join their affiliate programs. The vendors that agree and provide us the necessary technical instruments will be linked from the result page of Firewall Challenge.

Apparently Matousec has throwed away the previous appearance of independency and credibility, and took a clear choice for profits, regrettably money is like in many other cases all that count to him.

To me this mean that his firewall tests aren’t reliable anymore, and now another issue raise too: how to rate his previous firewall tests? Are they performed independent? Or is it possible that during these tests a conflict of interests was valid and therefore his tests are biased?

My thanks to Wilders/wat0114 for pointing out Matousec’s new policy.

When and where was Matousec born?

As far i know, on May 19, 2006 Matousec announced his birth and intentions for the first time on CastleCops, and to be sure to get the necessary attention he accomplished that aim via a spam post to reach optimal effect:

http://www.castlecops.com/pstp770180-.html#770180

His declared aim in that post was very noble:

I want to introduce a new security group to you. Our site is called Matousec – Transparent security and we are Internet and security related group of young people with desire in security research.Our first project is the Windows Personal Firewall Analysis and we have already published first results – a review of ZoneAlarm Pro. So, if you have ZoneAlarm installed on your computer you might be very interested in our review. We will continue with Sunbelt Kerio and others, read more.

It is surprising how fast his desire changed into what it is today…

Of course his spam wasn’t appreciated by the CastleCops folks, and very fast a reaction of them followed:

As a matter of ettiquete, matousec, it is considered very impolite to come to someone else’s site and pimp your own site in your very first post. In case you haven’t noticed, we have a fairly extensive review section on this site which includes a number of reviews of most popular firewalls, both pro and con. The discussion topics contain a lot of additional information as well.

Indeed Matousec’s post was very impolite, but he reached his aim: getting attention.

Starting from that day his star was rising, but like all stars, there is a day a star will tumble-down.
That fatal tumble-down event started today.

Update/Continuation 2009-05-03: Matousec’s New Moves to Recapture the label “Trustworthy”

Advertisements

April 20, 2008 - Posted by | Alerts, Downloads, Friends, News, Recommended External Security Related Links, Uncategorized | , , , , , , , , , ,

10 Comments »

  1. Hi Smokey,

    thank you for the acknowledgment, but Outpost Firewall Support Forum member/moderator minoka actually deserves credit for pointing this out in the moderator section of the forum 🙂

    Regards,
    wat0114

    Comment by wat0114 | April 21, 2008 | Reply

  2. Hi wat0114,

    but i traced your thread on Wilders so i give you the credits.

    Edit: “Now some extra facts about Mr Matousec’s Firewall Earnings Challenge, found today this on the Agnitum Outpost Personal Firewall Forum —- ”

    redrawed these facts, with thanks to Trel on DSLR.

    Comment by Smokey | April 21, 2008 | Reply

  3. The day an “independent” reviewer makes “recommendations” based on affiliate schemes, is the day they lose all credibility.

    The top performer in the Matousec tests was actually Comodo Free, so any legitimate review should have that as a recommendation. Strangely, Mr. Matousec has often seemed to be biased in favour of Comodo, wonder how much they were paying 😉

    The only way Matousec can recover credibilty, is to immediately cease to call these “recommendations”, and to site link all that fulfil the scoring criteria, even if there is no affiliate backhander.

    Comment by Matthew | April 21, 2008 | Reply

  4. Smoke, Very well written article and this
    trail has been crossed by a few folks trying to get to
    the bottom of Mr. ‘David’ Matousec
    I say good luck..it should be fun 😉

    Comment by hayc59 | April 23, 2008 | Reply

  5. Oh my God!
    What is this about?
    Did Matousec changed his tests to hide some features from the public?
    Does he not publish his test and metodology anymore? Does he not allow any reader to reproduce his results with any product and test on his own?

    Or are you just envious? If not, kindly prove that anything from above is valid or apologize. Otherwise stop speaking about someone’s impolitness!

    Comment by jp | July 16, 2008 | Reply

  6. JP,

    i hardly can’t believe you don’t know/understand what is all about, please read my article and the comments thoroughly and you will understand why Matousec’s “Firewall Challenge” is matter of discussion.

    Comment by Smokey | July 18, 2008 | Reply

  7. […] April 2008 I already blogged about wrinkles in Matousec’s Firewall Challenges. Instead of taking serious notice of my critism, he decided to ignore entirely my prewarnings. That […]

    Pingback by Matousec and his Firewall Challenges Hall of Shame 2008 Awardee « Smokey’s Security Weblog | November 30, 2008 | Reply

  8. If it is indeed true, are there any other more reliable independent testers out there whose objective is solely security/reliability and not profits?

    Comment by JC | January 6, 2009 | Reply

  9. If you can reproduce all his results, what’s the problem?

    His methodology is open and his source code for all his tests are available for noncommercial or private use.
    => http://www.matousec.com/projects/security-software-testing-suite/

    How hard is it to download the source code (which also includes the pre-compiled binaries), compile it, and verify the results yourselves against Matousec in a public forum?

    There’s no problem as long as you can reproduce his results with the PDF reports he has generated with each firewall he has tested.

    On the other hand, if his tests are closed and there’s no way to reproduce his results, then I’d agree that this credibility is in question. (I’d be very skeptical and suspicious if I can’t verify the results myself).

    Its a simple scientific approach: If you can’t reproduce what he has done, its BS. If you can, then there isn’t much to argue about, is there?

    Comment by aussiebear | January 18, 2009 | Reply

  10. […] will probably remember the critical article I wrote about Matousec and his Firewall Challenges, “Matousec’s Firewall Challenge wrinkle: conflict of interests?” and the honor I granted him to add his Challenges to “Smokey’s Security Weblog Hall of […]

    Pingback by Matousec’s New Moves to Recapture the label “Trustworthy” « Smokey’s Security Weblog | May 3, 2009 | Reply


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: