Christmas is different this year

With Christmas on our doorstep once again, it's time to spend time with friends and family, enjoying a well earned break and some festive traditions. This year, some of those traditions have changed.

All the good stuff is still there for us to enjoy. It's the only (Christian) period of the year where every house is festooned with decorations, a tree, lights, and a feeling of cheeriness. nearly every meal is indulgent, nearly every drink is mulled.

In years gone by, these delights, and the enjoyment of the season was offset somewhat by the unpleasant side of Christmas. The crowded shopping centres, the battleground of supermarkets and the sheer effort of getting ready for the big event.

"Having a list... checking it twice" was not the job of Santa anymore, but of diligent parents with a family to buy presents for, and at least a few crucial meals to plan. Did you buy gardening gloves from Waitrose for Granny last year, or the year before? How old is niece number 2 now, and is she a fairy princess or a climbing trees type of niece? 

But in today's world of Amazon Prime... or even eBay Free&Fast... much of the drudgery of Christmas shopping has been consigned to memories of yesteryear. 

Making good use of Google sheets, sync'd across all my devices, I was able to keep track of the 30+ presents that needed buying, and with mobile apps for Amazon and eBay, plus mobile-optimised webshops becoming the norm, every spare moment was an opportunity to work through the list, and get everything ordered. Always for next day delivery, and as such, so much easier to track.

Working out what to buy was made much easier through Whatsapp messages with each of my children. Even Granny has got the hang of emailing links to present ideas on the web. In fact my family shares ideas throughout the year, mostly through whatsapp links - which are often discussed around the dinner table afterwards. And to my surprise, a lot of the messages are more along the lines of "This would be so cool for Grandad", rather than "I want this for Christmas".

I've also retained my self-appointed title as savviest shopper. For every present a price comparison is a few clicks away. Finding a great present, and being sure you've paid the best price, without cris-crossing a shopping mall over and over is delightfully satisfying.

Unlike fresh fruit or groceries, present buying is perfectly suited to the online experience. In grocery shopping, some control of quality and quantity is handed over to the store ... think choosing a ripe avocado, or receiving a mini shampoo bottle because you didn't notice it was 50ml not 500ml. At Christmas, present buying is a breeze. Assuming you stick to trusted brands, getting the right product is not dependent on whoever is picking that product at the warehouse. In fact, the process is mostly automated. Taking humans out of the loop is a very good thing.

This year, in particular, I've been somewhat housebound on account of a recent hip replacement and a doctor-enforced 6 week break from driving. But has that ruined my run-up to Christmas? Not a jot. In fact, it's given me the ideal opportunity to test-run a 100% online Christmas. And it's been a breeze.

As long as Ocado arrive on 23rd December, as promised, and as long as they haven't run out of parsnips, my Christmas is in a perfect state of readiness. In fact, it was that way by December 4th. Since then I've merely been seeing even cooler presents, and replacing them on my list.

It's a rosy picture but not without some downsides. The boot of my car, and also my conscience, is full of Amazon cardboard and packing paper. Amazon seems incapable of delivering more than one product per unnecessarily large box. I ordered three wall calendars, 30x30x1cm. Each of them arrived in an A3-sized 15 cm deep box, filled with packing paper. The sustainable forests that produced that cardboard, wept.

I'm also worried about the last mile distribution. I'm on first name terms with a variety of delivery drivers. Most of whom have been kind enough to mention how well I'm doing, having started to receive deliveries whilst propped up on crutches, to now being crutchless. But all of them were driving diesel vans, mostly old ones, often leaving them running whilst dropping the parcel. I imagine the current push for cleaner air in and around London is being set back considerably by this.

In years to come, I'm hoping that big data analytics can be put to task to optimise these delivery runs, whilst electric vehicles eventually replace the oil-burners. Hey, even drone deliveries might make a dent, if they become safe and cost effective.

So, small quibbles aside, I'm looking forward to a mostly stress-free holiday period having got ready for it with the least wasted effort, the most efficient e-commerce, and all well within budget. That all makes for a very merry Christmas.

Will the £1BN spent on G.Fast get us back to "TV which actually works" ?

Slightly behind the curve on this story, as a result of getting a hip replacement a week or so ago. And that will be a critical point to remember as I rant my way through the next 300 or so words.

After coming down from my lovely morphine-induced high I left hospital with a shiny new hip and a bagful of painkillers. Like many men, my pain threshold is rather low and, as such, I was in a foul mood for a little over a week.

Things that contributed to my foul mood:

- BT TV not working 
- BT Broadband not working
- BT Home Hubs not working

... and when I say not working, I don't mean not working at all. What I mean is that these services work fine most of the time. In fact I'm quite happy with my suburban FTTC service at around 36 MBPS. 

But what got me so riled up, whilst spending 14 hours a-day confined to the sofa was the internet news telling me that we'll soon have gigabit internet through G.Fast, whilst simultaneously experiencing a fibre broadband service that randomly stops working.

TV used to work. You turned it on, and unless there was a howling gale, it worked. All the time, with no drama. No troubleshooting, no settings, no routers and no wi-fi. 

Now we have a vastly superior delivery mechanism for glorious streamed HD content, which occasionally just stops, you know, for a minute or two. That, to me, seems a backwards step. It's a good job the emergency services don't rely on broadband, or 4G for that matter. Oh, wait... they do.

If the government wants to invest £1BN in something, it should invest at least a little of that in a proper Ofcom group to assess the reliability of the current BB & FBB end-user service that customers of BT and its competitors are actually offering.

In the same way the EE wants to shift focus from population coverage to geographical coverage. I'd like Ofcom to switch from speed and coverage measures, to "does it work all the time" measures.

Everytime you see a message saying "connection lost", "download failed" or "not connected to the internet". That is clear evidence that your provider has dropped the ball. Count up those occasions, for everyone in the country, and Ofcom would have a decent idea of how reliable each service provider is.

Come on Ofcom, I know its important to tinker with BT to find the best ideological division of its groups, but what we consumers care most about is whether what we've been sold, and what we're paying for actually works. Not some of the time, all of the time.

And when I'm still on the maximum allowable dosage of codeine & paracetamol, I care about this A LOT!

My Telco is doing WHAT with my data ?

Telecom companies have some of the richest, deepest data sets of all industries. Because the business model involves contracting customers to monthly plans or providing pre-pay SIMs which people rarely change, the Telcos understand their customer better than almost all other industries.

You might buy a car once every two or three years, or take a holiday a couple of times per year. That gives the automotive and travel industries a little data about what you like, and what you might buy next time. But Telcos have access to a constant stream of data from your smartphone, set-top TV box and internet connection. 

The only industry that gets close to the level of data that Telcos have is Retail. The supermarket loyalty card has revolutionised the retail experience. Anyone who has had a book of vouchers through the post from Tesco, and been delighted that there are special offers on exactly the products they buy frequently, has been touched by this new data-driven approach.

Telcos are in a race to offer 'quad-play'. That is home phone, mobile, broadband and TV... bundled together from one supplier. In UK you can choose from BT, TalkTalk, Virgin or Vodafone. And it's not just the extra revenue they will get from you from each service that interest them, it's your data that is the gold mine they are prospecting for.

So what data do Telcos have on you? Well, the answer is a little disturbing. If you take all four services from, say, BT. Then BT can extract and store your data from each of their systems. Those systems contain records of your calls, your TV viewing habits, your location and your internet activity levels. With a little extra effort, they can extract details of: everything you've watched, live and on-demand; whether you watched the ads or skipped them; what you google and which websites you visit; where you travel frequently and which route you take.

If that wasn't creepy enough, they can pull data together from each of these systems and identify a profile based on other people's behaviour to understand even more about you. The observations that they can make are sometimes based on probability, but the accuracy of the profiling is enough for the Telco to make better decisions on how to keep you as a customer and sell you more services.

Here's a very short list of some of the things they know about you, or could derive from multiple data sets.

Sounds a little scary... right? Well, it's not as bad as it seems. For one thing, the Telco companies are not terribly interested in you. As long as you pay them your monthly fee or top-up regularly, that's their main concern. They might try to sell you some related products like more TV channels, cloud storage or phone insurance. They'll use the data for that. But most of the value in your data is in selling it to other companies.

Just in the TV data which Telcos have access to, there is a revolution underway. TV viewing ratings used to be collected by panels of viewers with special equipment. In UK the BARB panel was around 5,000 households. All the TV ratings were extrapolated from this small group of panelists. Now, every TV box can record every program watched, every button press, every play/pause/record action. All this data is sent back to the Telco in real-time. This gives highly accurate real-time data to share with advertisers, to share with production companies, and to add to all the telephone, mobile and internet data the Telco holds.

The system works so well that leading TV providers are able to insert targeted adverts into your live program feed, which specifically target you as an individual. They know it's you watching, and not your wife or your kids, because the way that you use the remote control buttons is as unique as your fingerprint.

Telcos have a system called DPI (Deep Packet Inspection), which allows them to look inside the IP packets which make up all internet traffic. DPI data shows exactly which web pages you've visited along with time-stamps, details of cookies, usernames (not passwords), and purchases. 

So combining this with your viewing behaviour, the Telco can tell a TV advertiser exactly whether their TV advert directly resulted in a visit to their site and a purchase. It's the most information an advertiser has ever had about the effectiveness of their advertising channels. And for that, they pay handsomely. Even basic personal data is worth a few pence per customer in the multi-billion pound data brokerage business. If that customer is in a high earning job, or high spending period of life, that value rockets to around £1. Multiplied by the millions of customers that Telcos have, this represents a tidy earner for the Telcos. And that is yesterday's model. Today's real-time model changes the game. 

In India, an insurance company is working with a Telco to give away free life insurance. In a market where only 3% of the population has life cover, there is a huge opportunity. The life insurance company has seen that it can get detailed lifestyle data from the Telco that it previously couldn't find. In poor communities, families don't have regular income, bank accounts, credit cards or even loyalty cards. But they all have a mobile phone. The mobile phone is the only way to get information about this sector of the population. 

Of course, the insurance company is betting on a proportion of those taking the free basic life cover to upgrade to a paid life cover. With the mobile phone derived customer data, the insurance company can make accurate cost/benefit analysis on each application for paid cover. Previously the risk was too great and the cost of cover too high. Now, the risk is calculable and the cost can be tailored to the individual.

Telcos are, in general, sensitive to the power they have with all this data. There is often strong governance about what data can be used within the Telco and what can be shared outside the Telco. As the intelligent systems being deployed by Telcos start to use these data sets, your experience with the Telco should improve. They won't post you as much junk mail, or spam you with as many irrelevant offers. They will be able to fix network or device problems before you're even aware of the problem, and they should be able to tailor your service to your needs at a competitive price.

But if that doesn't settle your nerves, there is European legislation coming in 2018, which will allow you to ask any company to delete all the data that it holds about you. Simply and quickly, you will be able to revoke their access to your data. When UK leaves the EU, it's anticipated that we will have similar legislation too.

And if you are very worried about what your data says about you, then I'd advise you to clean up your act. The data reflects your reality. If your reality is such that your data is, at best - embarrassing, and at worse - incriminating, then you only have yourself, and your data, to blame.

The search for Smarts part 2

Since my original post 'The search for Smarts', my Garmin Fenix2 came back to life. 

The achilles heel of Garmin's chosen charging technology is the contacts on the rear of the watch. With occasional muddy/sweaty use virtually guaranteed, the contacts can misbehave. Without being visibly 'dirty', the contacts can pick up some kind of oxidation, which doesn't allow the watch to charge through the charging clip. That results in a dead watch which you think is fully charged.

To fix the problem simply requires a wipe of the contacts with an abrasive fabric, wet-dry paper or the serrated edge of a coin, and the watch will happily recharge.

So, that's what happened and my Fenix2 is back to normal and my search for a new smart watch is on hold again.

In the intervening period a report was released about the state of the smart watch market in 2016. Despite the impressive marketing budgets behind many smart watch offerings, the market has declined by 51% year-on-year for the quarter. The figures were heavily driven by Apple and Garmin.

Q3 2015 was the first full quarter that the Apple Watch Series 1 was commercially available, and it took almost 70% market share in that quarter. The Apple Watch Series 2 was only available in 2 weeks of this quarter, so it's too early to say whether the 72% decline in Apple Watch sales will be offset. 

Garmin is the run-away winner in the segment, shipping 5x more watches in the quarter than in that quarter in 2015. The sport-specific smart watch market which Garmin dominates is a play-ground of micro-niches which Garmin understands and has tailored their bewildering array of watches towards. Whatever kind of athlete you are in your free time, Garmin has a watch for you. The Apple Watch, however, is a one-size fits all offering. (ok, two-sizes, but you get my point). Your Apple Watch doesn't mark you out as a triathlete, an ultra-runner, a gym-goer or a mountain climber. The Garmin on your wrist leaves no-one in any doubt which sporting preference you have.

And here's the key point in the decline of the smart watch market. Your watch is a functional accessory. It isn't your means of communication, your window on social media, your media player or your entertainment whilst commuting. It sits on your wrist as an overt display of your life-style preferences. There is a reason why people chose to wear a Rolex or a Casio, and that reason is 'difference'. The non-smart watch market is well established, with a huge range of options at both ends of the price scale. Are you a Gold Rolex or a Patek Philippe kind of guy? Are you a Casio CA53W or a Casio F91W guy? Up until mid 2015, nobody was an Apple Watch guy. And now, most people who wanted to be an Apple Watch guy already are. Everyone else is not-an-Apple-Watch-guy.

I would love to see Apple and their smartphone competitors continue to develop the rather impressive technology in smart watches over the coming years. They need to do more to tempt people away from the simplicity of a Casio, or the prestige of a luxury watch. 

I'm an CA53W guy, in case you were wondering. 

For or against the pay-wall

The Telegraph's online news site was my go-to as a counterpoint to the near ubiquity of the BBC here in UK. I was perfectly happy with The Telegraph, and would put up with attention grabbing adverts, advertorials, promotional features, because the news I wanted to read was free.

Today, The Telegraph online news service introduced a 'Premium' service, asking £2 per week for access to this 'Premium content'. At first glance, it seems that anyone with an opinion, or with a sense of humour, or even simple story telling chops has been put behind this pay-wall.

Flat news is important, and can be glanced over to keep abreast of happenings. For those subjects about which I feel a deeper level of interest, I want to know what other people think, I want to be entertained, and I want to have my views challenged. All of this, on The Telegraph, is now behind the pay-wall.

I should mention that the adverts, advertorials and promotional features are still there. 

I recently switched from Chrome to Firefox, an inferior browser with a quirky attitude to hyperlinks, purely in an attempt to remove the epidemic of almost full-screen automatically placed ads. I was fed up of scrolling multiple times to get to the meat of the articles in pages full of inappropriate, repetitive, invasive ads.

The Telegraph's paid-for apps for phone and tablet also never appealed to me. Perhaps because most of the content was available for free through the online web service. They are marginally more appealing now, but only because the online experience is now unpleasant.

Sure, I could spend £2 a week to 'unlock' Premium content, or I could go the whole-hog and drop £6 a week on Premium with Digital - and have the same content nicely arranged, with no ads in a swish app. But here's the problem...

I don't want to pay now for something I used to get for free. I'm looking at a Telegraph news page where nearly half of the articles (and 90% of the ones worth reading) are locked. As a non-suscriber, I don't want to be shown everything that I can't have, as a constant reminder that my purse-strings are pulled too tightly. I want to have what Telegraph provided for me for years, for free. If they're not going to do that, I'll go elsewhere. I'm sorry, Graham Norton and Simon Heffer aren't enough to keep me loyal. 

What I'll do instead is go to the trouble of configuring Google News with my interests, dig deeper into BBC News, and cherry pick from Huffpost, The Guardian, and more specialist press - all of which are free-to-me.

So sorry Mr Chris Evans (Editor not DJ)... you've lost me, possibly forever. If too many others fail to immediately re-value their Telegraph news feed - by £104 per year, the Telegraph will have created a bigger problem than falling newsprint customers. They'll rapidly fade into online obscurity.

Rely on google for travel planning at your peril

Today's mission, should you choose to accept it, is to get from Ascot to Paris... without drama.

"I accept", I thought to myself, and immediately jumped onto Google to devise a plan.

Google says it'll be between 25 and 45 minutes drive from Ascot to Slough station, at that time of day. Arrive in time for the 17.10 express to Paddington. Five stops on the underground, and glide into St Pancras International with 45 minutes to spare before the 19.01 Eurostar to Paris.

You've got to love the Open Data concept making all this timetabling information available. I do also love Google maps for tracking every journey unobtrusively and delivering by far the best 'Sat Nav' I've ever used.

Confident of a solid plan i left the house on the dot of 16.20, leaving time for the Legoland traffic, parking, paying and getting to the platform. Leaving plenty of time for the express 18 minutes to Paddington giving 45 minutes before the Eurostar was due to leave. So far, so good.

Cue the crackly voice over the tannoy.. "We regret to inform passengers that the 17.10 to Paddington has been cancelled".

Oh crap.

"Passengers wishing to travel to Paddington, please go to platform 5 for the stopping service leaving at 17.14." A stopping service that takes 38 mins, NOT 18 mins.

Ok google, what do you have to say for yourself? Lulling me into a false sense of security, assuming UK trains run when they should.

Don't panic reader, my 2012 New Year resolution comes up trumps again. That year I decided to be everywhere 15 minutes early - on top of any contingency needed. That one resolution has single-handedly reduced my daily stress ten fold.

So, with the 24 minute train delay, minus my 15 mins, I arrived at St Pancras 6 minutes before the check-in closed.

Now, here's a suggestion, Google. We all mostly understand probability. Why not take the published reliability figures for every train service and overlay a UK-rail-delay-modifier?

"The likelihood of this train running on time is 87%. Your arrival is expected at 18.15, but you may arrive 24 minutes later. You can save 24 minutes and take the risk, or have a 99.5% chance of arriving by 19.39."

For casual beers-with-friends arrangements, I'll take the risk. For a connection to Eurostar, a customer meeting, or a West End show, I'll play it safe.

The data is there to be used. The algorithm is trivial. So, come on Google... we're relying on you now!

And I shouldn't have worried. The 19.01 Eurostar was delayed anyway. Of course.

Tesla's level 5 autonomous driving and why I nearly fell over

Self-driving cars again, huh? So much noise on that channel, and here is Tesla knocking out news that isn't news. All the new Teslas will be equipped for level 5 autonomous driving. They're at level 3 right now, where a human has to sit behind the wheel and be able to take over at any time. Level 5, however, is 'no human needed'.

Of course, the level 5 isn't yet activated, because that's not allowed yet. But it'll soon be painfully obvious that cars are better than people at driving, and the legislation will adapt

Yep, before long you won't need to drive. You could read a book, take a nap or even not bother going and let your car do the work.

Two and a half days waiting for your kids?

And thats when I nearly fell over. I was on my way to collect my middle child from an after school club when it hit me. With this new Tesla, I could just send the car. Hold the phone!

The lives of those of us fortunate enough to be burdened with one or more progeny are blighted by the need to ferry young ones hither and thither, as we attempt to enrich their out of school learning. Football practise, music lessons, school trips arriving back late, play dates... you get my drift.

But here's a Tesla that could do that for you. Send the car, stay on the sofa, and enjoy a restful evening. Hey! Why not just give the kid the Tesla app and let him send for the car when he needs it.

The Tesla is a better, safer, less impatient, more focused driver than I am anyway, so why take the risk of driving myself. If Granny wants to pop over and Grandad doesn't want to drive her, send her the Tesla. She can even tune it to Classic FM if she wants.

I want level 5 and I want it now.

But wait. Aren't those trips in the car some of the only times you get to chat with your kid one to one? Those precious 15 min therapy sessions, locked in a moving car with no blaring screens and no headphones in ears? The delighted retelling of a successful sporting conquest, the sharing of angst and frustration from a particularly bad day, the reminiscing back to past times when an 'oldie' (from 2011) comes on the radio.

I hope that Tesla and their self-driving-car  competition don't rob us of our last precious moments of shared parental time before the kids themselves have learnt to drive, or learnt to Uber, or learnt to do whatever it is they have to, to get around in the next decade.

In the meantime, I'm going to relish these pick-ups just that little bit more.

No VR until this...

Despite the huge buzz around VR, I'm not convinced, and I'm not buying. Oh... and no one else is buying it either.

VR has, and always has had, a big problem. It requires you to put something on your head that makes you look ridiculous. It closes you off from the world around you and leaves you oblivious to real world events.

The only place that most people would accept these problems is in their own bedroom or, more likely, bedsit. There will always be a good number of early adopters who will buy the next new thing, for gaming and for ... erm... well, thats it. Just gaming.

The day I buy VR is the day that the headset is a pair of glasses. Not dopey google glass glasses, but a regular Ray-Ban or Oakley. Because we're so very good at miniaturisation, it wont be too long before your regular sunnies will happily overlay real life with an HD screen and enough battery life to give you a couple of hours of entertainment.

Black Mirror's video-recording eye-balls made compelling viewing, and brought about some fun discussions about the moral aspects of video recording. It rather skipped over the aspect that i found most appealing... the ability to lie down, press a button and watch your own personal entertainment, without finding the best seat on the sofa, the best viewing angle and the perfect sized screen.

Put the screen in the kind of glasses that we actually want to wear and the nature of entertainment will have been changed dramatically. Only recently have we unplugged the TV to carry it around with us. The next step is to wear it, everyday, everywhere we go.

Brexit... it's Y2K all over again

Brexit will be as big a challenge as Y2K was back in the late 1990's. 

The threat of 'The Millenium Big', where computers just weren't ready to handle 4 digit year codes was an all-consuming passion for CIOs and CTOs of the day. Estimates put the cost of fixing the Y2K bug at between $320bn (IDC) and $600bn (Gartner). Brexit could, for Europe and UK, be a very costly exercise.

The European General Data Protection Regulation (GDPR) due to come into force on 25th May 2018 specifies the way that organisations should handle the data that they hold. This extends to consumer rights to transfer their data from one company to another and to delete all data held. Organisations will be held to task with fines up to €20m for poor data management and data breaches.

When the UK invokes article 50 of the Lisbon Treaty it will have 2 years to complete it's 'Brexit' of the European Union. No country before has left the EU, and only one (Greenland) has left the block, then the European Economic Community (EEC).

The proposed Repeal Act proposed by Theresa May will define the strategy in law for removing UK from the European Communities Act 1972. A number of current EU laws will be adopted by the UK on 'Day Zero' to simplify the exit. But many of those are likely to be altered in the 2 year period following.

As a separate entity outside the European legislative area, UK will not be subject to the GDPR and it is thought that there may not be a 'safe harbour' agreement in place protecting the use of UK data within the EU. Companies operating in both UK and Europe, or just in UK with customers, partners or suppliers in Europe will need to rapidly consider measures to protect computer assets and data.

Companies will have to consider systems segregation between UK and EU, which will almost certainly require separate physical instances to be created. Data may have to be repatriated across the UK/EU divide, to ensure it sits in the right legal framework for its use. Processes which span national borders will have to be overhauled, often with the advice of legal experts. Revised trading and tax rules between EU and UK will have to be built into systems, and businesses may have to re-engineer their operating models so as not to be penalised.

All in all, Brexit is going to be costly for any organisation with an interest in or dealings with Europe, and for IT players, just as with Y2K, there is an opportunity for short-term windfall projects to manage the changes.

Whether you are for or against Brexit, it's going to create sizeable opportunities for the IT firms who can capitalise on the large-scale changes that Brexit will bring.

Hadoop, huh! What is it good for? Absolutely... everything!

You may come across Hadoop in your everyday tech-world job, and you might even understand what it is. But one thing's for sure, people in our business tend to get very excited about the next new thing.

A few years ago, Hadoop was the next new thing. Google open-sourced it's GFS code in 2003, and Apache began to develop the HDFS and Map Reduce ecosystem. The promise of handling petabytes of data was an intoxicating elixir to anyone sitting on a lot of data. 

By 'a lot of data', we don't mean some big financial record files, or customer databases, we mean BIG. Start from 4 terabytes, because below that your normal file system, Microsoft or Oracle relational database can handle it - on a server with a bunch of disks attached. 

4 terabytes is a lot of data. It is equivalent to the data held in 132,000 regular 500 page fiction books. Stack those books up and your pile would be 8 miles high. Dig down, and 8,000 miles gets you to New Zealand. And that's where Hadoop gets going.

In 2010 Facebook was using 2,300 Hadoop clusters, which can all work together, to store 40 petabytes. Now a petabyte is 1,000 terabytes. That's a pile of books which will almost get you to Mars. So you can see that this really is an astounding amount of data. 

But it's not just words on a page, or lots of numbers that are being stored in Hadoop. The Hadoop File System is great at holding data of all sorts. Where in a regular relational database, you need to know what you want to put into the database before you put it there, Hadoop is like a magic dumpster. You just throw anything you like in there and worry about getting it out later.

I say it's magic because, unlike a regular dumpster, the data doesn't decompose over time, and it can be sorted and 'mapped' to help you find what you're looking for down the line.

But, don't think of Hadoop as either a Dumpster or a Library of books. Think of it more like a shopping mall. It's somewhere people can go to find what they are looking for. Sure, you need an idea of what's there before you start looking, and you might need the help of a mall map. But everything about that mall can be uncovered. Whether it's a comparison between price-tags on similar t-shirts, how many lattes Starbucks sold yesterday, or what the teenagers hanging around in the parking lot are saying to each other. Any and all kinds of information can be stored and retrieved.

This is why it is so compelling for companies to create a data lake, with as much data in it as possible. Where previously your departments held the data and didn't share it, now everyone is sharing all their data, and the correlations, relationships, inaccuracies, trends and insights are ready to be discovered. 

Imagine if a retailer in the mall wanted to know who was buying their products, who was buying competitors products, at what price, and what they were saying to their friends about it, and which shops they visited before they made the purchase, and when they were going to come back, and what would make them choose that store? 

It would be an unfair advantage. And that's what companies get from Hadoop and their data lake. It's an unfair advantage over their competitors who don't have it. The treasure trove of Big Data is hard to fathom, and even harder to implement, but will prove itself over time to be the best way to understand and run your business.

Mourning the death of a true friend

The 28th September 2016 marks the end of an era. The announcement by Blackberry that they would cease production of mobile devices brings to a close one of the most compelling narratives of the turn-of-century tech battles.

Blackberry, then known as Research In Motion (RIM), launched their first mobile device in September of 1996. That was the month I started my first post-university job at mobile phone competitor Ericsson. Back then, mobile phones were just that. Phones that made calls, analogue calls at that - with the TACS network, providing good coverage but poor quality voice connections. Using a 'mobile' for anything other than talk and texting was a distant dream. A dream that almost nobody had yet had.

Blackberry's entry into the pager market was remarkable mainly because it was one of the first devices to provide 2-way paging. Until Blackberry arrived, your pager buzzed and you found the nearest telephone to respond with. The Blackberry 900 allowed for a text response, and was rapidly adopted by professionals with a need to be immediately informed and provide a response within seconds.

With the launch of the Blackberry 7100, a monochrome GSM phone, the inclusion of enterprise-ready email was a revelation which catapulted RIM into the number one spot for companies looking to mobilise their workforce and management. The simple UI, the jog-dial, and the full qwerty keyboard gave execs a headstart in being always-on and in control of their business.

These companies' IT departments were also delighted to have a single server installation to slot next to an MS Exchange or Lotus Notes server, which handled the mobility, the security and the load from the rapidly growing usage of mobile email. As I recall, the Blackberry Exchange Server (BES) was available as a perpetual license for just £25,000. Pretty cheap, even for 2002.

Adding a colour-screen, multimedia handling and a browser into ever more compact devices ensured Blackberry grew at pace through the early 2000's. It's end-to-end security and multi-national network gave governments and large corporates the confidence to adopt the service wholesale, and for a few years the competition failed to get close to their dominance. Nokia, Ericsson and all the others had a tough time competing in the enterprise world. It was only the coming of Apple's iPhone in 2006 which upset the Blackberry apple-cart.

The glory days of happy Blackberry users cossetting their Curves, Pearls and Bolds lasted between 2007 and 2010. My Curve 6310 was the best device I've owned. The Bold 9000, the second best. If smartphone usage had stopped at calls, messaging and emails, then I'd still be a Blackberry fanboy today. But it didn't, and I'm not.

During the 2007-2010 period Blackberry themselves had diversified into 'Pro-sumer' devices for the mass market. The Blackberry Pearl, in 2007, became the device of choice for teens and those on a budget. Blackberry's own messaging app BlackBerry Messenger (BBM) was unique in that it wasn't charged by the message, as SMS texting was. Those with a tight budget flocked to use BBM, from chatty teens to illicit traders. The latter group benefiting hugely from the end-to-end encryption and the anonymity that BBM provided.

Aside from the phenomenal success of the iPhone, Blackberry had other challenges. The qwerty keyboard, so necessary for the texting/emailing brigade got in the way of providing a rich multimedia experience. Blackberry produced a bizarre half-way house product, the Blackberry Storm, to compete with touch-screen iPhone and Android competition. But the 'Touch & Click' experience of the Storm was woeful. Despite a hugely expensive launch and marketing budget with Vodafone, the Storm flopped. 

Then came the false-starts. Blackberry's first tablet, the Playbook, was well specified but ran the new Blackberry Tablet OS based on the QNX Neutrino OS which Blackberry bought in 2010. It was destined to become the cross-device OS to replace BBOS, but the lack of a strong app development community and the predominance of Apple and Android in the app market dealt the death blow.

During this period Blackberry had suffered heavily publicised system failures and security litigation. In September 2011 Blackberry suffered a 72 hour outage which prevented service in most of Europe and parts of US and Asia. It was caused by a combination of a Cisco switch fault, an Oracle database corruption and the ensuing traffic backlog. Blackberry users, so wedded to their instant email, internet and enterprise apps were stuck without even a basic service. For enterprises this is a major inconvenience, but for the emergency services, security organisations and many others, this was unacceptable.

Blackberry also received heavy criticism for it's handling of police requests, both in UK and India, to unlock the end-2-end security of the service. The London riots in 2011 were subsequently found to have been, in part, organised through anonymous groups of criminals using BBM as their primary communications method. India, aware of this growing illicit use of the service demanded that Blackberry open up their system to government tracking and Blackberry went to court to prevent that, losing any goodwill that they had left among governments around the world.

Subsequent device and OS launches were received with muted reviews, many harking back to the great Blackberry devices of days gone by. OS 10 now supported Android apps, the Priv and Passport devices found some die-hard customer support, but the end was nigh.

With the recent announcement that Blackberry are moving away from device production, I'll mourn the loss of a true friend. Those Blackberry Curves and Bolds saw me through my first ten years of employment, enabled my mobility and responsiveness. The devices, which sat so comfortably in one hand and the jog wheel by which I scrolled through my emails on the tube, will be sadly missed. 

Time and tide moves on, and Blackberry didn't. Like IBM's OS2, Betamax & HD DVD, Blackberry devices have been consigned to the history books, un-edited Wiki pages and the bottom of desk drawers. So let's bow our heads in a moments silence to honour the passing of a true friend.

Analytics for trains

Like Jeremy Corbyn(sic), I was consigned to the floor of a Cross Country train for 3.5 hours today. For a sizable fee of £175, I was locomoted between Reading and Manchester in some discomfort. And it really gets you thinking.

How can a fixed capacity train service be sold to overcapacity? The demand is of course peaky, very peaky. But then so are networks of parcel distribution (at Christmas), mobile phones (on new years eve) and swanky bars (on a Friday night).

Despite the advanced analytics on offer today, these capacity-critical services all seem to apply the same rule... "Deliver the worst service we can get away with".

Mostly this is driven by good business sense, but also by pure supply and demand. Train Operating Companies (TOCs) are contracted under the 'Cap and collar' model whereby if they lose money the government bails out a good proportion of their losses or, if they .ake money, takes a cut of their profits. Profits or losses vary widely across the rail network. In 2015 South Eastern Trains amassed £803 million in profit. Paying their shareholders a dividend of £25 million. And this was achieved with an average seat occupancy of 26%. Imagine how their profits would look if they could bump that up to 30% of even a third.

The problem here is that the contracts TOCs sign include a minimum service provision, which forces them to run off-peak services at a loss while trying to make up for that by stuffing peak time services to the gunwales. This means misery for daily commuters and the occasional traveller-on-a-schedule.

But is there a better way? Erm... have you been on a train in Germany? German trains run like clockwork and everyone has a seat. And, the cost of this experience ? Well.. that's where the problem lies.

Germany spends €17bn on rail subsidies per year compared with €4.5bn in the UK. That extra €12.5bn is equivalent to the UK government spending across roads, buses, trams and railways combined.

If we want a pleasant rail service we either have to pay for it, at the expense of the NHS, social security, defense, education or pensions, or we'll have to accept a service which more closely matches supply with demand.

Very basic analytics could allow us to optimise the off-peak services. For example, instead of providing time-boxed services (e.g. hourly), we should work on functional services (e.g. demand-driven). This might mean a train service between towns and cities twice or thrice in a day.  We could gradually reduce the public expectation of trains running when we want throughout the day, to having a reduced but reliable and comfortable service in the mornings, middays and evenings.

Sure, people would have to organise their activities differently to adjust to this new paradigm. But that, to me, is progress and good money sense.

So let's tear up the TOC contracts and deliver some societal change in the form of a better rail service for all.

The search for Smarts

I'm in the market for a smart watch. After 4 years of loyal service, my Garmin Fenix 2 has expired, leaving me with only two other ways to track my running and cycling activities.

But rather than mourn the loss of a dear technological friend, I feel it's only right to commence the most comprehensive evaluation of all possible replacement devices.

Which watch? 

I run, cycle, work out, and occasionally swim. So speed and distance measurements are important, as is heart rate monitoring. I couldn't give two hoots about stride length, cadence, vertical oscillation, VO2 Max or recovery periods. I'm not Mo Farrah. Oh, but it has to Bluetooth to my  phone, obviously.

So I set off on my epic campaign to review all the information available to a would-be watch buyer. And, wow! What a choice there is. Even if I stick with good 'ol Garmin, there are 28 different models to choose from, not counting the different colour schemes.

Nearly all of them do what I want. And they range from the £849.99 Fenix Chronos, down to the £89.99 Vivofit. Garmin's UK website doesn't allow filtering by functionality, only by 'activity', and not by more than one activity at a time. I don't want to be pigeon-holed into just being a runner, or just doing 'tactical training', whatever that is.

The only other option is to compare devices. There's a limit of 4 devices on their comparison tool which, given I haven't narrowed my selection down from 28 yet, seems inadequate.

I'm done with the Garmin website. Next up is the excellent www.dcrainmaker.com. DC's website gives phenomenal in-depth reviews of most of the latest sporty watchy gear. He leaves no stone un-turned in the pursuit of feature and performance testing. Both the watch and the software get a thorough going over, and useful comparisons against other devices and his annual favorites. Some of the reviews run to 'dissertation' length. And honestly, I don't have the time!

I checked out his '2016 Best Buys' list, and found the Garmin Fenix 3 topping the list, alongside the Garmin 920XT. Looks like a simple choice... I liked the Fenix 2, why not buy the Fenix 3 and get on with my life?

BUT WAIT....

What about a proper smart watch... you know, like the Apple Watch, but not so w[SENSORED]ky ?

Since I bought my trusty Fenix 2, the market has exploded with smart watches from all the phone companies, most of the watch companies, and a host of other people too... Mont Blanc eStrap-on? Huh?!

And do I want a smart watch, or just a smart strap? Or connected trainers? Or just use my phone - which does everything a smartwatch does better.

I'm going for a lie down. This is all too much. I pity the general public in making these kinds of decisions, except for the ones who like making comparison spreadsheets, and the ones who just buy whatever's on offer in Argos. For the rest of us, it's time to embrace the impending shoppers' remorse, and 'chose-the-wrong-device' anxiety.

Now... where's my spreadsheet from 2012 ?

IOT or IORT (Internet Of Rubbish Things) ?

The Internet Of Things (IOT) is coming! Along with 5G, SDN & NFV, the IOT has been driving industry article column inches for a couple of years now. And what do we have to show for this IOT hype? To most people's minds... not a lot.

IOT breaks down into a few areas: wearables, automotive, home, medical, industrial, transport. And reports suggest anywhere from 10 to 50 billion 'things' will be embedded with the ability to connect to a network. That's 5 to 10 times more devices than there are smartphones on the planet.

A year ago I thought those claims were wildly off the mark. No one really needs a bluetooth toothbrush, and all those internet-connected fridges simply haven't happened. So I lay back and imagined that the IOT was another industry fad, consigned to promise much but deliver little. This isn't the IOT ... the Internet Of Things, it is the IORT ... the Internet Of Rubbish Things.

In the history of our telecoms world, there is an enjoyable tendency to get over-excited about something new, then forget about it and focus on the next 'something new'. And then, a few years later, let the world catch up and hey-presto... that promised 'something new' finally happens, to dramatically change the world, but without too much fan-fare

I was at the Mobile World Congress in 2007 the year after iPhone launched, and remember the buzz around 'Mobile TV'. I remember the quirky Japanese phones with screens that rotated on their unwieldy soap bar handsets to show video in landscape mode.

There was an expectation that, with the advent of 3G, video and TV services would launch across the mobile carrier world and be the next big growth driver for data usage. Vodafone duly launched video-calling and showed at least one Cricket test series on their free trial of mobile TV. 3 Months later the trial had ended with no commercial proposition resulting. 2007 happened to be the year that Netflix began it's streaming service and began it's move away from posting DVDs to the public.

Cut forward 9 years, through 4G, fibre-optic, wifi-n, and we arrived at the mobile-video-enabled Rio 2016 Olympic Games. Almost half of all content streams from Rio 2016 were made to mobile devices.

This dramatic change of behaviour is what we expected in 2008. It just took a little while to get there. To go from a failed mobile TV pilot to this point, where mobile video is something that not-just-millenials have wholeheartedly adopted, indicates what the future may hold for IOT.

IOT is best when the customers don't notice it's there. "Any sufficiently advanced technology is indistinguishable from magic." So said Arthur C Clarke. And this is where IOT is the magic.

These tiny embedded sensors that track your movement and heart beat, and share them with your smartphone. The SatNav in your car which shows live traffic information. The thermostat that warms your house when it sees that you're coming home. All happening magically without us having to think about it.

So the IOT revolution will happen not with a fan-fare, but with a billion high-frequency whispers. And before you know it, everything will be connected, and the 'everythings' will be making your life that little bit better.

The challenge of Authentic PR

A recent Telecom.com podcast featured Paul Nolan from CC Group, who quote-un-quoted "Authentic" as a recent buzzword too often used by PR agencies. Equating it to IOT, future-proof and 5G-ready - the current favorite buzzwords of the Telecoms business.

This made for a fascinating comparison between industries. In the Telecoms world we have no end to the deluge of latest innovations and the buzzwords that accompany them. In PR, it seems the diatribe is also alive and kicking. 

I quite enjoyed seeing that the world of PR has just as many infuriatingly meaningless terms as Telecoms. Have we had enough hotly-anticipated, paradigm shifting, market-optimized, horizontal, well-positioned, marquee solutions...?  Probably not.

Trevor McDonald, our favorite barrier-breaking newsreader, would be turning in his grave over the lack of plain English and specificity displayed by so much of the Tech company PR content.

But, to my mind, "Authentic" is what we should be shooting for. We should all be authentic in our interactions with other humans. As Patti Azzarello suggests in her excellent managers book 'Rise', being authentic is fundamental to developing effective relationships.

So let's all agree to put a little more authenticity into our communications, whether we're lowly middle-managers or high-flying PR execs. We'll all be better off, in time, and better informed, in information.

... and if you are in the business of writing copy for yourself or your business, here are some great guidelines taken from Apple's approach to clear, compelling messaging.

Who do you trust ? Tesla autopilot .. or .. the distracted teenager behind you ?

Following on from today's post on driver-less cars, I remembered watching a great TED talk by Jennifer Healey at Intel.

She points out that a human driver has two eyes with which to scan the surroundings, and those eyes are not designed for the task. For starters, they only see clearly in one direction. To check behind, the driver stops looking where he/she is going. Not looking where you are going whilst driving at motorway speeds surrounded by other large metal objects is frequently fatal.

So what if cars could talk to each other and say things like... "I'm slowing down rapidly" or... "I'm changing lane". And other cars could listen and respond immediately and in a safe manner. That sounds like a better plan that relying on my two eyes and poor reaction time, or even worse, the concentration and lack of experience of the distracted teenager in the car behind me.

And Jennifer points out that only cars within say 100 meters of each other would need to talk, so they wouldn't require a cellular connection, just a white-space radio messaging function.

Today's more advanced cars use arrays of sensors to detect their environment and they do a pretty good job of looking at the car in front or behind. Top-end cars already have traffic sensing cruise control. But each car is an island on its own, detecting it's own necessary actions. With car-to-car communication, a car two or three cars behind on the road would already be taking action before the car immediately in front of it had reacted, cancelling out the 'traffic wave effect'. 

It's simple, it's intelligent, and because it doesn't require expensive arrays of sensors in every car... it would be cheaper and more accessible to lower-end vehicles.

Let's hope the auto industry take a pragmatic view of these possibilities and deliver a solution in the not-too-distant future.

References:
Jennifer Healey - If cars could talk, accidents might be avoidable
CDC - Teen Drivers: Get the Facts
New Scientist - Shockwave Traffic Jams

Driverless cars? UK maybe, India no hope.

The buzz around driver-less cars is hotting up. Suddenly Google are being made to seem slow-to-market by Tesla and Uber. It seems the machine-driven car is here already and aside from a few years of human-aided operation, 'the machines' will have a big impact on our modes of transport.

I never understood why trains needed drivers in the first place. They run on tracks, the junctions are controlled by points operated by machines, and there is minimal traffic to worry about. I suspect it may be customer fear and major reluctance from laggard trade unions that have restricted the automation of what seems to be a much simpler transport problem. At least we are seeing a gradual introduction on newer lines (Docklands Light Railway) and some older lines (Victoria line), albeit with human 'train captains' still on board to handle emergencies.

The easier train lines to automate are new ones, which don't suffer from the same key problem as driver-less cars do. The problem is humans. Wherever automation and humans need to coexist there is more complexity and a much higher degree of risk.

This is the key problem when putting driverless cars on the roads. It's not the computers that are the problem but the illogical, unpredictable, and frankly hazardous humans.

On 30th June 2016 Tesla reported that it had made the NHTSA (National Highway & Traffic Safety Association) aware of a fatal accident. A Tesla Model-S collided with a tractor trailer on a divided highway as the tractor was crossing perpendicular to the road. The Tesla autopilot failed to identify the trailers white side panels against the sky and did not detect the chassis of the trailer. This tragic event left a family without a father and served as a stark warning to the driver-less car industry.

Even in countries with established and, by-and-large, good adherence to road safety and conduct rules, there are between 5 (UK) and 12 (USA) deaths per 100,000 vehicles. that works out to 3 (UK) and 7 (USA) fatalities per billion km's driven. The Tesla fatality (US) was the first in over 200 million km's, indicating a better safety level than that of human-driven cars. 

In India 130 people die per 100,000 vehicles. In Bangalore, where I find myself this week, 2 people die on the roads every day. I wish I'd have known this figure before I rented a motorcycle for the day. I might not have been so keen.

The reasons for this alarmingly high death toll are numerous. There is appalling traffic congestion, there are overcrowded and poorly maintained pavements, there are no traffic lanes, huge numbers of scooters and small motorcycles. The clearance between vehicles is often inches rather than feet. Nervous drivers go nowhere, and traffic signals are like the pirates' code - more guidelines than rules. Add into the mix a few thousand tuk-tuk taxis, poorly maintained vehicles and roads, animals running loose in the road, and you need a vastly more complicated compute model for driver-less vehicles.

If the Google/Uber/Tesla software wants a real test, they should be (and probably are) testing here in India. If they can drive us safely in Bangalore, Mumbai and Delhi, then Los Angeles and London will be child's play for 'the machines' to navigate safely.

Net Neutrality vs Utilities

We often hear that Internet Access is now considered a basic human right, and should be thought of in the same way as other utilities such as water and electricity. Dumbing down the argument to this level is fine for the tabloid press, but let's take a more detailed look, and decide for ourselves.

What is net neutrality? It is the principle that ISPs and governments should treat all data the same, not discriminating or charging differentially by any one of many factors e.g. type of content, which website is being visited, what device is using it and for what purpose.

Most ISPs and mobile carriers have the ability to throttle (i.e. reduce) speed and bandwidth in their network, in order to provide reliable service, even-out peaks in traffic and provide a better service. If you use a 4G mobile phone, your voice calls are converted into packets of data and will more than likely have used the internet to send those packets to another 4G mobile user that you are calling. And yes, those packets will have had priority within the carrier network, to make sure that the call doesn't break-up, and provides real-time two way conversation.

What carriers are not allowed to do is to treat Voice IP traffic from competing companies different to their own traffic. They are also not allowed to throttle traffic that they don't like the look of. Peer-2-peer file sharing networks which carry a vast amount of illegally shared content can't be throttled in preference to perceived 'legitimate' traffic.

Without net neutrality, ISPs could deliver tiered services, to content providers in addition to those tiers offered to the consumers. So a provide a real-time video e-healthcare services (think remote midwifery), could secure a high quality video feed, whilst an e-book download service could secure a cheaper less performant access because they don't need ultra-low latency and high bandwidth.

Why net neutrality is GOOD:

1. Equal access to the internet is a human right. The internet has been a mostly free and open place for all of humanity. Charging "different rates to different website" would destroy that principle

2. It is good for competition and innovation. It protects smaller companies from the leverage of larger companies who could pay more to deliver content and services.

3. It fosters fairness in ISPs, who currently are paid by consumers to provide access. Without net neutrality, the ISPs could charge content providers for priority delivery, limiting the consumers' overall level of service.

Why net neutrality is BAD:

1. If ISPs can charge more from content providers, they could improve the network faster and provide better overall service, depending on the traffic being carried.

2. Other parts of the network already deliver tiered services across the internet. Amazon charges more for faster processing and better storage, which already impact on the consumers' experiences.

3. Many consumers can choose between Broadband providers with competing networks. In UK, between BT hosted services and Virgin's cable network. If consumers don't like how their ISP is providing differentiated service, they can opt to move to a competitor.

Conclusion

Internet access is 100% not like a utility. No one is offering faster, better electricity. It's all you can eat electric until a circuit breaker blows. You can fill your bath over and over, and nobody has better or more water to offer. This is an infrastructural limitation. There would be little point laying two sets of water pipe, or two electric lines into a house.

However, the analogy breaks down a little when you consider that it's perfectly possible to buy extra water to drink when the need arises, or special water for a special occasion (Perrier anyone?)

Internet access is the same. We can choose how to access the internet, be it at home, work or mobile. We already pay differentiated access fees. But would we suffer if say Netflix could pay more to TalkTalk to secure a buffer-free experience for its users? 

I suspect that we would. Corporate decision making is seldom driven by what's best for the customer. If an edge can be gained over a competitor, then a large company typically will do what it takes to get ahead. Those competitive decisions would have the potential to limit choice, increase cost and stifle innovation in the services that we consume.

A policy for a government not to pursue net neutrality would have to be regulated by even more complex limitations. Sticking with net neutrality seems to be one control mechanism that protects the consumer, and simplifies the governance of internet access to all.

Reference material:

https://en.wikipedia.org/wiki/Net_neutrality

http://plugandplaytechcenter.com/2014/05/16/four-and-against-the-net-neutrality-debate/

http://www.callforwarding.com/blog/net-neutrality-voip-need-know/ - Caitlin Burge