Simonseeks Blog

Keep up to date with the most current News, Travel Tips & Developments from Simonseeks with the new Blog.

Simonseeks rankings: what you should know

Thu, 03/12/2009 - 14:53 Posted by Rick Madden

Simonseeks has both Guide Rating and Writer Ranking systems. This article is concerned with Writer Rankings (or Simonseeks Rankings).

Every day people upload travel guides to Simonseeks and interact through comments on blogs or guides, sharing their passion for travel. In turn, you, the community, acknowledge the most active and talented community members through guide ratings and marking people’s comments as useful (or otherwise!). We value this contribution and want to acknowledge it in the Rankings, hence the term Simonseeks Ranking is more appropriate than just “Writer Ranking”.

We also know people are trying to trick the system, thinking that by viewing their guide lots they will move up the rankings and earn more cash – you know who you are! This isn’t the case, and never was, but we want to do all we can to ensure the Ranking is a fair and consistent measure of a member’s contribution, quality of writing, popularity within the community and other important factors.

So what’s changed?

The Simonseeks Ranking is a complex algorithm, which we’re not about to give out, that takes into account more than just views. Some of the factors are easy to figure out, for instance the higher the guide ratings the better, however others are less obvious; How many comments have you added? How many of those comments were found useful by others? How does your guide rate against the site Bayesian average?

How will this affect me?

The new Rankings have already been updated, you can go to your profile to see your current position or look at the list of Writers to see how you stack up against others (this currently just lists our 600 or so writers but will soon become a full list of our 5000+ community members).

Do Simonseeks Rankings affect payments?

No. Well, not yet anyway ;0)

What’s the benefit of a high Ranking?

Let’s be honest, most people like to rank highly within their fellow community. We also believe the higher your Simonseeks Ranking, the more trust a reader will have in your guide/s.

But it’s not all about stature, sometimes it’s just about good hard cash, and as Simonseeks becomes more successful we’ll look for ways to reward members for their contribution. We’re in the process of giving away 20 grand in writing prizes to our community.

How do I improve my Simonseeks Ranking?

Just get involved. Write guides, rate guides, add comments, rate comments, check out the blogs... it all helps. But whatever you do, don’t try and trick the system! Feedback from the community (such as guide ratings and comment helpfulness) plays a big part in determining you rank. Writing dozens of guides may help in one respect, but only if they are good. Adding hundreds of comments is great, but you want people to vote them useful. We can also spot vote stuffing (rating your own guides lots) and it can work against you.

How often are the Simonseeks Rankings updated?

Rankings are updated both daily and monthly depending on the ranking factors we’re using. You may notice small changes throughout the month but the biggest shuffle is likely to occur about once a month - we'll tweet about it to keep you informed.

I can’t understand my Simonseeks Ranking position?

Don’t worry if you can’t figure out why your Ranking is exactly where it is. Instead, focus your energy on getting involved, contributing good quality content and making your guides as inspirational and informative as possible - see the Editor's blogs for tips from top earning guides and ideas for new guides.

But I still don’t understand????!

Don’t worry, you don’t need to!
 

Comments: 30
Rick Madden's blog

Community comments (30)

Fri, 26/02/2010 - 11:54 Posted by Allie Reynolds (Moderator)

I agree with Tim, Kevin and other contributors that have expressed their confusion in terms of the Simonseeks ranking system. I understand that this is in its infancy, but I also think that some changes are definitely merited.

In the top 20, there are just 4 writers with 5 or more guides, while there are 9 writers with just one guide. Inspirational, talented and loyal writers who have consistently contributed to the site such as Anthony Peregrine, Lee Marshall, Annie Bennett, Natasha Edwards, Kevin Hughes and Richard Field – have contributed over 10 guides each, have nearly 30,000 views between them, have average guide scores of over 4.1, and yet none appear in the top 25 rankings?

Out of curiosity, I googled “Bayesian averages” (really sad!), and as far as I can see the objective of this is as follows:

“to reduce noise due to low sample counts. In effect, the less votes an item has, the more it is pulled towards the average rating of all items. This is a mathematical term that calculates a rating of an item based on the “believability” of the votes. Items with less votes should count less than items with many votes and we can trust that this is how the public feels about it”

This means, the more votes an item has, the higher the “weight” of these votes. In applying this to Simonseeks, this seems to suggest the exact opposite of current trends in the ranking system.

Just a few thoughts but:

Should the ranking system accurately reflect the very best writers overall as voted by the public, community and the Simonseeks team collectively?
Should loyalty, overall contribution be weighted highly in terms of number of quality guides written, average overall score ?
Should earnings be taken into account?
Should celebrity / professional / enthusiast writers be ranked the same?

I think that is this is given some more careful thought and discussion, and that it is structured in such a way to promote quality and to incentivise contribution, this will go a long way towards encouraging existing writers to contribute more, and others to start writing.

Thanks

Fri, 26/02/2010 - 12:29 Posted by Rick Madden

Four great questions raised by Allie... what does everyone else think?

Here are my thoughts:

1. Yes - but should it only include writers?

2. Two seperate factors here: Quality and Contribution. Which is more important?

3. Personally, I do think success should be a factor - but I expect some interesting answers to this one!

4. I'd prefer one Community Ranking with the ability to sort and filter it so I could see 'All Writers', 'Celebs Only' or 'Top Commenters'.

Let's keep this debate going! Its great to get some feedback (writers and non-writers) so that we can factor this into the next revision of the rankings.

Rick

Thu, 04/02/2010 - 18:12 Posted by Tim Scrafton

Hi Rick,

I think the rankings system requires further discussion and perhaps another close look at how it merits.

BTW - I do not think it is a good idea to split writers into categories, and this to me would damage the concept irrevocably. This would in effect create a 'third class' Simon Seeks citizen out of your loyal 'travel enthusiast' contributors, many of whom would become defectors I believe. There are far more travel enthusiasts than either travel professionals or celebrities, and this type of site requires both lots of content and traffic.

Regarding Simon Seeks writer rankings...

It is not hard to work out that the - average views per guide - has a large bearing on the ranking position - however this unfortunately can then reward less content being generated by writers -

...the quality/quantity issue should not even be of concern as one of the main attributes of Simon Seeks is that it uses professional editors to make sure every guide has enough merit to be worth publishing, and in any event they are initially rated by the editors.

As a 'User Generated Content' site the 'rewards' or rankings should reflect the 'input' and usefulness and inspiration to travellers who come to the site for insightful info -and reciprocate this by giving the writers some degree of appreciation for their efforts, whether this be commercial, or via a degree of recognition provided by the ranking system, which to be worthy has to make sense.

If writers all submitted one guide each, it would be an easier matrix to work out.. however this is not the case. It would appear that more work generated by individuals can actually 'dilute' the success criteria when it comes to ranking position, which actually defeats the object.

I have been evaluating the site since I began submitting my ski guides in October, and it has to be said a lot has been achieved in the relatively short time the site has been live, and teething problems are part and parcel of any new project.

I will submit another blog to ask some questions regarding puzzling statistics, relating to the rankings.

Cheers Tim.

Fri, 05/02/2010 - 00:01 Posted by Rick Madden

Hi Tim, valid points.

Firstly, as far as looking more closely at the system, I think I have said somewhere else in the comments that this is only in it's infancy - I've plenty more ideas to implement yet, each ensuring a more fair and accurate way of ranking community members.

As for splitting into categories - I think the idea is more about maintaining one central ranking but allowing people to filter on traveller type to see who the highest ranked pro, enthusiast or celeb is (personally, I'd like to see a list filtered on enthusiasts as I prefer reading their content).

AVG views per guide. True, views are one element of the equation which I've been monitoring myself. I think it's weighted too heavily at the moment and that's one of the first things to look at when we get back under the hood. However, the more guides you write... the more contribution your make... and that does your ranking no harm.

Quality/Quantity - that's a whole separate conversation Tim!!

UGC/Rewards - yup, that's the idea.

More guides dilute the result - well, you've written 26 guides and are in 11th spot. So, aside from the views point covered above, there's the answer to that one ;-)

Evaluating the site - cheers, all feedback welcome and very much appreciated.

Hope this has clarified some of the points.

Like I say, plenty more work to do on this but let me assure you it's entirely intended to reflect a users contribution, popularity and recognition within their simonseeks community.

Rick

Mon, 08/02/2010 - 12:32 Posted by Tim Scrafton

Hi Rick,

Thanks for the detailed reply and feedback.

I note that you kindly left some comments and feedback reviews on a couple of my ski guides so thanks for that btw - (also as you are a fellow Manc - please let me know whether a 'bertie' or a red - so I know we are on the same wavelength !

Re:-in your reply...

"More guides dilute the result - well, you've written 26 guides and are in 11th spot. So, aside from the views point covered above, there's the answer to that one ;-) "

- As I mentioned in the previous blog - I have some extremely confusing stats regarding the rankings and performances pro rata to highly ranked writers who only have 1 or 2 guides published. I have been compiling these daily for sometime - and the results don't actually bear that out... it will be interesting to see what you make of them...

I will post them up later today hopefully.

Cheers Tim.

Mon, 08/02/2010 - 12:48 Posted by Rick Madden

interesting... email them to me at richard.madden@simonseeks.com. I'm dying to see what you've been up to!

ps: there's only one team from manchester ;-)

Wed, 17/02/2010 - 13:16 Posted by Tim Scrafton

Hi Rick,

Just a brief example of how the rankings are weighted too much in favour of average views - per guide -

I have done a comparison below, but have no issues with the writer in the example or the quality of his work.

#3 ranked Michael Richardson has written 1 guide, and taking that example and comparing it to one of my 26 (my most viewed guide) over a 10 day period, his guide was viewed 46 times, while mine was viewed 229 times. I was ranked #11.

Then taking in the bigger picture including ALL his guides which is the '1' mentioned and all mine which is a further 25 guides - I had 1050 views versus 46 views over the same 10 day period.

This is a 22X bigger margin and surely a demonstrably higher value to Simonseeks?

I was ranked #11 at the time, I am now ranked #8, having added another guide and another 359 views since that period... to the 29 recent views of the consistently ranked #3 which remains exactly the same?

I find this algorithm to be bewildering in how it incentifies contribution...

I could understand there being a table ranking the most viewed guides - which you have, but to acquire an overall ranking surely there ought to be a more rounded balance to adequately reflect work input and TOTAL number of views... and number of guides.

..currently the 25 guides extra - I have input over higher ranked writers such as #3 with just 1 guide published, in effect renders the 8000+ more 'views' worthless in the ranking order.

Will this be taken into consideration when you next look 'under the hood'?

Any feedback would be appreciated.

Cheers Tim.

Wed, 09/12/2009 - 17:45 Posted by Elaine Housby

I totally agree with Kevin's comments. Including celebs in the ranking is a cynical marketing ploy and an insult to serious writers. Like him, I have no interest in earning money from this site, and in any case, it's obvious that it can only ever generate token pennies which are not worth worrying about. I wrote my guides because I loved the places and wanted to tell people about them. I know that watching your ranking is a bit childish but it is fun and the only real reward you get. I had crept up to 243 and then suddenly shot up to 50 while the new system was being worked out. I got all excited because I thought this meant you liked me. Then once the new system was implemented I had gone down to 286 which is the lowest I have ever been. I wouldn't have minded so much if part of this hadn't been to make room for Cliff Richard. Also, I would like to think that I got some credit for commenting on other people's guides - I went to the trouble of looking up and reviewing all the guides on New Zealand and North East England, which are the areas I've been writing about, but this doesn't seem to count as a contribution to the site which affects ranking. The upshot of it all is, I am almost certain, that what is rewarded on this site is not the quality or interest of the writing but the number of hotels puffed. This to me explains why some writers with average reviews are highly ranked, because readers rate them for providing interesting information about a destination while the editors rank them for mentioning lots of hotels. I think this jeopardises the USP of the site which is supposed to be to provide completely honest appraisals of accommodation personally stayed in, because there is a clear incentive both to list lots of hotels, when you can't have stayed in more than one at once, and to emphasise the luxury end of the market, because the payment is a percentage.

Thu, 10/12/2009 - 12:01 Posted by Rick Madden

The number of hotels reccomended IS NOT, has never been and never will be, included in the ranking algorithm.

Fri, 04/12/2009 - 11:40 Posted by Rick Madden

Just to clarify... comparing one writer with another is interesting and can be fun, but it doesn't tell you the whole story. The algorithms look at individual performance in relation to the whole of the community.

Google bayesian averages and ranking methods for more. Anything on ELO Ratings might be worth a read also.

Thu, 03/12/2009 - 21:56 Posted by Kevin Hughes (Moderator)

I have to say I care and I‘m disappointed and more than a little disillusioned with Simonseeks following the introduction of the new ranking system.

Naturally we all aspire to climb as high as we can up the rankings, that is human nature.

However, how is it we suddenly find Sir Cliff Richard ranked 1 and another celebrity writer, Raymond Blanc, in 2? Sir Cliff has written two guides and Raymond Blanc one. Then we see professional writers of the calibre of Trevor Claringbold relegated to 7 despite his 37 guides, 7,000 plus views and 23 guide comments. The superbly talented writer and travel enthusiast Allie Reynolds drops down to 11 despite her 19 guides, 9,550 plus views and 25 guide comments. It simply doesn’t make sense.

Personally I dropped from 9 to 73. Now I was flattered at being so high and never thought I should have been anywhere near 9 but it certainly gave me some encouragement.

As for trying to trick the system as I have dropped so far I presume I am considered to be one such writer. I freely admit I have checked how many views I have had on my guides within My Workspace but how else can I check? If you don’t open the review you can’t find out. In any case I never thought those views were counted.

Personally, I couldn’t care less about earning money from my reviews I just want to share my experiences and simply love writing reviews. I object to any suggestion I may be involved in trying to earn more cash by fiddling the number of hits on my reviews.

In view of this, the £200 prize I am to receive from one of my reviews in October I wish to be donated to the Countess of Chester Hospital’s Relative Comfort Appeal. If it is to late to divert this money to the appeal before it reaches my Paypal account then I will personally deliver it along with a member of the Simonseeks editorial team.

Like many other writers I just want fairness. The writer rankings have been up and down since the site was launched and in my view celebrity writers should not even be included as editors always rate them with full marks and they are bound to attract a large number of hits.

Please sort it out once and for all.

Thu, 03/12/2009 - 22:54 Posted by Rick Madden

Hi Kevin, appreciate your views.

To try and explain why Cliff and Raymond are at the top of the rankings (without giving the formula away)... these guides are very popular, which admittedly is down to the PR they have received. However, they are also very highly rated. For example, Trevor and Allie do have lots of good guides but they are on average rated lower than those of Raymond and Cliff - Cliff (4.4) and Raymond (4.2) are very good ratings. Of course as mentioned above, this isn't the only factor taken into consideration and as you will notice we have other celebrities who are not ranked as highly.

Your own situation is most definitely not to do with any alleged vote stuffing activity - checking and editing your guides is absolutely fine and is encouraged.

I can assure you the new method is by far fairer than any previous formulas, however it is entirely dependent on the community getting involved. This is why it is so important that the community really get behind the good writers and rate their guides higher than those you don't think are as good.

Re the celebrity writers involvement, the Ranking system is only in its infancy and we have lots of ideas for refinements. For example, we are considering adding a filter so that Professional Writers, Enthusiasts and Celebs can be listed separately (as you suggest).

Hopefully this allays some of your concerns. I shall forward your donation gesture onto the Editorial team who will be in touch.

Fri, 04/12/2009 - 10:30 Posted by Kevin Hughes (Moderator)

I am sorry Rick but I cannot agree. Firstly, under your new system the top five ranked writers have between them produced just eight guides. While I agree we should be looking for quality above quantity there are a number of very good loyal writers who have put a great deal more into this site.

In the top 10 we have three celebs who have produced just four guides between them, and lets be honest, if they weren’t celebs they would be not be where they are.

Your argument that Trevor and Allie have lower average guide scores than Sir Cliff and Raymond fails if we then look at Catherine Lawler who is ranked 15 with one guide, which has an average rating of three and has not been commented on at all, compared to the previous top ranked writer under the old system, Tim Scrafton, now ranked 23 who has produced 21 guides, 47 comments and has an average guide rating of four.

Further I never suggested in my first blog that professionals, enthusiasts and celebs should be listed or ranked separately. What I did say was celebs should be separate as they are inevitably given a high guide rating by editors and will always attract higher viewing figures.

I do not believe travel professionals and enthusiasts should be ranked separately. As a journalist I have had travel pieces published in a number of newspapers and magazines across the UK. However, I class myself as an travel enthusiast not professional as travel writing was an aside to my normal work.

Whether your ranking system is in its infancy or not is irrelevant. What writers want, and I am sure this blog will attract many more comments, is an open, fair and above all, transparent system. What is currently happening is the writers who believed in the concept of Simonseeks and have so far put the most into it are, to some degree, feeling let down, confused and alienated by this ranking system.

Wed, 17/02/2010 - 00:56 Posted by Tim Scrafton

Hi Kevin,

I think you hit the nail spot on.

And your gesture regarding your prize only underlines your integrity.. well done!

Cheers Tim.

Sat, 05/12/2009 - 14:56 Posted by Siobhan Lillis

I have to say that I'm with Kevin on a number of points. I actually moved up the rankings under the new system, which was nice to see, but am still confused by it.

There seem to be a number of writers in the top 25 who have produced perhaps one or two guides with an average rating of three who rank higher than highly-scored, highly-viewed writers with multiple guides. The rankings do, after all, appear in the 'writers' section of this site - perhaps if other factors are going to be taken into account it should be renamed as 'community ranking' or similar.

Having said that, aside from this oddity, I really like - and am glad to be part of - simonseeks!

Fri, 04/12/2009 - 11:01 Posted by Rick Madden

There are also many examples where writers were previously lower in the Rankings than they are now. Tim Scrafton is a good example, as he was previously 262 in the rankings and is now 23rd (he was No 1 for a just a few days whilst we implemented the new formula).

At the end of the day, the power is in your hands! How many of these writer's guides have you rated?

Fri, 04/12/2009 - 11:01 Posted by Jon Ryan (Moderator)

I'm with Kevin on a number of his points, particularly the "open, fair and above all, transparent system" comment.

I posted a comment before about my perception of a "mummy knows best" attitude in some aspects of Simonseeks. I have asked for some specific information about just how my Symi guide earned the magnificent sum of 2 pence last month - why can't I just be told?

Yes the site is new and of course will have teething problems, but honestly people, for a site whose raison d'etre is the dissemination of information...

PS: Would anyone be interested in starting a Facebook site for Simonseeks contributors? May be a better place for these discusssions.

Fri, 04/12/2009 - 11:20 Posted by Rick Madden

We keep the algorithm secret to prevent abuse.

The Facebook site is being done at the moment... I'll ask Charlie for a date on when it will be ready.

Fri, 04/12/2009 - 19:12 Posted by Jon Ryan (Moderator)

How much abuse do you think I'd risk for 2p?

Sun, 06/12/2009 - 13:33 Posted by Trevor Moore

Jon, for someone who claims to understand that Simonseeks is a new siteand therefore in the process of building a community, you seem to be particularly aggrieved that your guides have earned so little. Just tell me again how much you've earned. I didn't quite catch it the first few times you've mentioned it.

I haven't written a guide as yet but hope to do so in the new year, not to earn a fortune but for the pure pleasure of sharing my travel experiences. I can understand why the 'algorithm' for the rankings isn't released as it would be all too easy manipulate the results by those who care about their ranking ie. all of you responding to this blog post.

All of you need to get a grip and enjoy the site for what it is - shared inspiration.And Jon - cut out the sarcasm, it's getting old now.

Sun, 06/12/2009 - 15:40 Posted by Jon Ryan (Moderator)

You've entirely missed my point, I'm afraid Trevor. Have another read of what I -and the others who are discussing this - have posted!

Mon, 07/12/2009 - 10:18 Posted by Trevor Moore

I have read your and others comments but maybe just maybe I've been distracted by some of your less relevant comments (?)

So, set me straight, what was the the point of the following comments you've posted;

1) "How much abuse do you think I'd risk for 2p?"

2) "Any suggestions as to the best way to invest or otherwise spend my earnings of £0.02?"

3) "But I would be willing to offer the entire sum of £0.02 to whoever is able to get the best buy for this amount on their next trip. I propose that this be called the Simonseeks Prudence Award and that a suitable trophy be crafted, to be handed over by some luminary in an appropriate setting on an annual basis. I can see this rivalling the Oscars in a year or two."

4) "Actually, the `best buy for 2p` may be worth pursuing. It could catch on if the Sundays could be persuaded to run something, or perhaps radio: 'What is the best thing you can buy with the last 2p of your holiday money?' Try it on the marketing brains up there?"

Mon, 07/12/2009 - 11:55 Posted by Jon Ryan (Moderator)

"So, set me straight, what was the the point of the following comments you've posted"

Well, Trevor, the first three were jokes, the fourth, arising from one of them, is actually a suggestion that I think worth exploring. Hope this is clear to you now - and good luck with your own guide writing!

Mon, 07/12/2009 - 12:11 Posted by Trevor Moore

Is it the same 'joke' repeated? Just so I'm clear.

Mon, 07/12/2009 - 13:55 Posted by Jon Ryan (Moderator)

Clearly you find none of them very funny. Sorry about that.

Mon, 07/12/2009 - 14:01 Posted by Trevor Moore

Apology accepted. Thanks

Thu, 03/12/2009 - 21:02 Posted by Richard Field (Moderator)

I've gone down from 17 to 93 today, but as you say "who cares"?! It's the money that matters to me, although beating Cliff Richard would be nice, too!

Thu, 03/12/2009 - 21:21 Posted by Rick Madden

I'm sure you'll catch up soon... Cliff's been top of the charts for decades!

Thu, 03/12/2009 - 17:32 Posted by Jon Ryan (Moderator)

Any news of a forum for discussing issues? I have been logging onto my Lesbos guide a number of times a day to see if some problems it has run into have been sorted. This has also resulted in it clocking up 18 (to date) comments from awkward conversations between me and the people who are trying to fix the problems for me (thanks, guys!).

Not only is this distorting the real figures, the sight of all the back-and-forth stuff is hardly enhancing the look of the site.

Also, would it be possible to discount log-ons from the IP address of the guide owner? This would help to stop `stuffing` and prevent anyone with an overdeveloped curiosity being accused of it.

Thu, 03/12/2009 - 18:17 Posted by Rick Madden

News on the Forums... we'll have something in place early in the New Year (maybe not Jan but definately Q1).

Conversations distorting the comment figures/look of the site... I agree, but it is nice to see some interaction (won't be an issue once we have forums).

Discount certain IP addresses... maybe Big Brother is already watching you ;0)