NFC in Firefox OS

[Reposted from https://hacks.mozilla.org/2014/11/nfc-in-firefox-os/]

Firefox OS is being developed in an open collaboration with Mozilla’s partners and community. In that spirit, and over the course of over a year, Mozilla and Deutsche Telekom (DT) teams worked closely together to develop a platform-level support for NFC within Firefox OS. During that time, both teams had regular product and engineering meet-ups for the end-to-end development cycle.
From proposing the NFC API, to defining the overall architecture, to prototyping and completing a production-level implementation on shipping products, this collaboration model worked so well that it really helped showcase the power of the “open” (Open technology and Open contribution model) for pushing the web forward. After all, this is exactly what Mozilla and Firefox OS stand for.
In this post, we describe a few basics around Firefox OS NFC implementation.

NFC Roadmap

Currently in release 2.0, Firefox OS supports NFC based sharing of contents (contacts, images, videos, URLs), as well as wirelessly reading information stored in NFC enabled tags (tag reading). Our sharing use cases are compatible with NFC enabled devices from other OSes like Android/Windows, so sharing of these contents across these devices would work. Our NFC API (first introduced in v1.3) has been put to use for these sharing use cases in v2.0 with core apps.
The Overall B2G roadmap is available on the wiki.

WebNFC API

The Firefox NFC APIs allow Peer to Peer (P2P) communication between any 2 devices that support NFC Data Type Exchange format (NDEF). NFC passive tags that can present themselves as NDEF compatible tags can also be read and written to. Firefox OS’ NFC implementation is currently for certified applications only, but as stated above, will be opened to marketplace applications as the API is developed to cover more use cases and data formats.

An example using this API

The following does P2P communications between 2 NFC devices (from the NFC API docs on MDN):

// Utility Function for UTF-8 string conversion to Uint8Array. // Or ideally, simply add this to your webapp HTML to use NfcUtils: // <script defer src="shared/js/nfc_utils.js"></script> function fromUTF8(str) { if (!str) { return null; } var enc = new TextEncoder('utf-8'); return enc.encode(str); }   var tnf = 1;   // NFC Forum Well Known type var type = new Uint8Array(fromUTF8("U"));   // URL type var id = new Uint8Array(fromUTF8(""));   // id var payload = new Uint8Array(fromUTF8("\u0003mozilla.org"));   // URL data, with a record prefix 0x3 replacing http://   var ndefRecords = [new MozNDEFRecord(tnf, type, id, payload)]; var nfcdom = window.navigator.mozNfc;   nfcdom.onpeerready = function(event) { // event.detail is a session token var nfcPeer = navigator.mozNfc.getNFCPeer(event.detail); var req = nfcpeer.sendNDEF(ndefRecords);   // push NDEF message to other NFC device. req.onsuccess = function(e) { console.log("Successfully pushed P2P message"); }; req.onerror = function(e) { console.log("P2P push failed!"); }; };

More such examples that ship with Firefox OS can be found in Using the NCF API.

Current Supported data types

The WebNFC API currently supports NFC Data Exchange Format (NDEF). There are some future plans for Non-NDEF types. From the example above, it is 4 fields, which is defined with 3 optional Uint8Array data types. The TNF and type are used to route the message to the appropriate registered web application(s).
(Source: http://git.mozilla.org/?p=releases/gecko.git;a=blob_plain;f=dom/webidl/MozNDEFRecord.webidl;hb=refs/heads/v2.0)

[Constructor(octet tnf, optional Uint8Array type,  optional Uint8Array id, optional Uint8Array payload)] interface MozNDEFRecord { /** * Type Name Field (3-bits) - Specifies the NDEF record type in general. * tnf_empty: 0x00 * tnf_well_known: 0x01 * tnf_mime_media: 0x02 * tnf_absolute_uri: 0x03 * tnf_external type: 0x04 * tnf_unknown: 0x05 * tnf_unchanged: 0x06 * tnf_reserved: 0x07 */ [Constant] readonly attribute octet tnf;   /** * type - Describes the content of the payload. This can be a mime type. */ [Constant] readonly attribute Uint8Array? type;   /** * id - Identifier is application dependent. */ [Constant] readonly attribute Uint8Array? id;   /** * payload - Binary data blob. The meaning of this field is application * dependent. */ [Constant] readonly attribute Uint8Array? payload; };

Note, in upcoming Firefox OS releases, we will be updating the data types slightly to make TNF an enum type instead of an octet.

Mozilla’s Flame device supports NFC, more devices coming

Our Flame device supports NFC and we are expecting more commercial devices from our partners soon. Flame device supports NFC chipset from NXP (PN547C2).

Videos

Here is a demo video of some of the NFC sharing features based on Firefox OS:



Core Apps In Flame device that use NFC:

• Gallery
• Video
• Music
• Settings
• System browser

A sample 3rd party App

Here is an app that Mozillian Dietrich Ayala put together using the NFC tag reading API. BikeCommute is an app that registers an NFC tag to track bike commuters at the Mozilla Portland office. The app is running on a Nexus 4 with Firefox OS 2.2, and is built with Famo.us for UI and PouchDB for data storage and syncing to a remote CouchDB. Currently, the app just reads the user’s email address from a text record written to the tag.
The next version will add support for running the app on users’ phones, using a local contact (user) instead of a plain text record, and being able to configure the NFC tag from their own device. The plan is to develop leaderboards from the CouchDB data and Mozillians.org integration so we can deploy and compete with other offices and Mozillians everywhere! The source code is available on GitHub and pull requests welcome!

Here is a Video demo of this app in action:

More NFC documentation

• For Developers: NFC API details
• GSMA API 6.0 Compliance of FxOS (in progress)
• For OEMs: NFC Compatibility doc

So, there it is!
We are really excited to introduce this new addition to growing list of APIs and features in Firefox OS! We hope developers will take full advantage of all that NFC enables by way of device-to-device sharing and also services like contactless payment planned in future.

When can developers start using this API?

Currently this API is available for certified apps. We can’t wait to finish the work to make this API available for privileged apps, so all of you developers can take advantage of this. If you wish to follow along or jump in and help out, feel free to track Bug 1042851. We are targeting to finish the work for the next release v2.2.

Next in NFC

In upcoming releases, with the help of our partners, we are focusing on expanding the NFC coverage for supporting Secure elements and services like NFC based payments. More on that in a separate post later. Please stay tuned.
Here’s to the open web!

Enabling Voice Input into the Open Web and Firefox OS

 

With the advent of smartphones triggered by iPhone in 2007, Touch became the primary mode of input for interacting with these devices. And now with the advent of wearables (and other hands-free technologies that existed before), Voice is becoming another key method of input. The possibilities of experiences Voice Input enables are huge, to say the least.
They go beyond just merely interacting with in-vehicle devices, accessories and wearables. Just think of the avenues Voice Input opens up for bringing more technologies to more people. It’s enormous: Accessibility, Literacy, Gaming, VR and the list goes on. There is a social vibe there that definitely resonates with our mission at Mozilla detailed in our Mozilla Manifesto

How it started

Both current leading mobile OS/ecosystem providers of today- Apple & Google have their native experiences with Siri and “OK Google” (coupled with Google Now). We really needed an effort to enable Voice Input into the first ecosystem that existed – the open Web. Around MWC 2013 in Barcelona, when Desigan Chinniah introduced me to André Natal – Firefox contributor from Brazil, we had a conversation around this and we instantly agreed to do something about this in whichever way possible. Andre told me about being inspired from a talk by Brendan Eich in BrazilJS, so I did not have much convincing to do. :-)

First steps

We had numerous calls and meetings over the past year on the approach and tactics around this. Since “code wins arguments”, the basic work started in parallel with Firefox desktop and FxOS Unagi devices, later switching to Mozilla Flame devices over time. Over a period of the past year, we had several meetings with Mozilla engineering leads on exact approach and decided to break this effort into several smaller phases (“baby steps”).
The first target was getting Web Speech API implemented, and getting acoustic/language modules integrated with a decoder and giving that a try. Lots of similar minded folks in Mozilla Engineering/QA & community helped along with guidance and code-reviews while Andre moonlighted (on top of his day job) with a very high focus. Things moved fast in past month or so. (Well, to be honest, the only day this effort slowed down was when Team Brazil lost to Germany in FIFA 2014. :-)) Full credit to André for his hard work!

Where are we?

Our current thinking is to get a grammar-based (limited commands) app working first and distribute it in our rich & diverse international Mozilla community for accent-based testing and enhancements. Once we have this stablilized, we will get into the phase 2 where we can focus more on natural language processing and get closer to a virtual assistant experience sometime in future that can give users voice based answers. There is lots of work to do there and we are just beginning.
I will save the rest of the details for later and jump to the current status this month. Where are we so far?
We now have the Web Speech API ready for testing and we have a couple demos for you to see!

Desktop: Firefox Nightly on Mac

1) http://youtu.be/1nSUvZlLMt8
2) http://youtu.be/R2PPz-O93X0
Editor’s note: for full effect, start playing the two above videos at the same time.

Firefox OS demo

http://youtu.be/65WmRw46-1U

So Come, Join us!
If you want to follow along, please look at the SpeechRTC – Speech enabling the open web wiki and Bug 1032964 – Enabling Voice input in Firefox OS.
So jump in and help out if you can. We need all of you (and your voices). Remember “Many Voices, One Mozilla”!

 

My notes from CES 2014

My notes from CES 2014

I have been to CES a couple times in the past and I know it is a big show. But surprisingly, it keeps getting even bigger. After it broke record 130K attendees in past couple years, this year people expect it to be around ~150K when audits come in. What that means is every keynote worth attending takes at least a ~1.5 hour wait in line. Lines for lunch and coffee too. Even after my rigorous planning (demo/keynote timing, long walks between show floors etc.), I could only cover about half of my list. Nevertheless that was plenty to get a feel for what’s coming.

Here are the trends I saw and some thoughts at the end –

Wearables

Pebble Steele Smartwatch

This category was very hot this year - so much so that - there were almost no smartphone / tablet mentions this year in “best of CES” awards or even product announcements. Most of the new demo’s were about wearables. Some were cool like the Pebble Steel which definitely moves their smartwatch product from a geeky plasticky gadget-watch to a more mainstream/fashion accessory that regular people will feel comfortable wearing.  (Some other smartwatches were like wearing a big phone on your wrist. See Neptune Pine). Also, I saw several upgrades to current health / fitness bands, smart dog collars too.

LG announced LG LifeBand touch and heart-rate headphones. Intel was especially big on this trend. They announced a $500K funding competition “Make It Wearable” for startups/dev’s. Qualcomm already has announced their Toq smartwatch at Uplinq. It is pitched as a developer edition watch for now to push the category (and their BOM parts like mirasol display for it!).

Home Automation / Entertainment / Internet of Things

Intel based Baby Monitor

Chipset makers and OEMs like Qualcomm, Intel, Samsung, LG looked big on this, as they want to own this space - what with smartphones / tablets saturating. Qualcomm is also pushing for an open standard for software framework called AllJoyn in this space and had demos for several home devices “talking” with each other. Intel showed a market-ready baby monitor that has sensors for baby’s vitals and the alerts are sent to other devices at home (like mommy’s phone or even a coffee mug with a display). Intel also announced its Edison ultra-small computer that you can use to enable hobby-based home automation projects.

Intel Edison

Samsung announced their SmartHome service to control various appliances via Samsung phone, TV or Gear watch. Same with LG which allows to control appliances via SMS. Mantra here is: Buy our appliances and devices so you can control them all seamlessly. (Standardization, anyone?)

Sony’s multiple synchronizable (projected or mounted) displays demo (on wall, ceiling, coffee table) was impressive. You can use all of these displays separately or in a sync.

Sony "Throw" projector

Several use cases can be enabled by this. E.g. play “music to sunset” with all displays showing a relevant (different) imagery. Or play “street scene in Paris” in-sync on all displays. Impressively immersive, but this looked a bit expensive proposition for mass-market appeal but not too far away in future. Their futuristic interactive display projected on a coffee table (or a wall) that you can interact with (touch and move UI widgets) was pretty cool. 

interactive UI projected on a table

Wall display (mirror when off)

Ultra-HD TVs, 4K Content, TV/Social integration

Pixels have gone up to 4K and TVs can come in curved panels (switchable by user between flat v/s curved). They look really immersive. In the soccer match demo on Sony’s dual-4K TV, one could clearly see faces of players and stadium viewers. Also, with social integration, you can “watch TV with the world” with all the social feeds time-sync’ed with the content e.g. A missed goal in a soccer game and you can see a burst of reactions from twitter on a side panel, or in your wearable glasses.

Sony, Vizio, Samsung, LG are heavy on UHD TVs. Mostly planned in mid-2014. TV Sizes are going beyond 85-100inches now.

YouTube had 4K streaming (VP9) demos running at partner booths like Panasonic TVs. Netflix announced they will roll out the next season of “House of cards” in 4K. (I would have preferred the sequel of “Breaking Bad” in 4K, but I digress). And yes, Sony had (expensive) camcorders for 4K recording by consumers.

Car Automation / OS / Electronics

There was so much action around cars that CES felt like an auto expo as well. Intel has partnered with BMW i3 car integration. Qualcomm introduced an automotive chipset 602A (Quad core, 320 GPU, bunch of multimedia / gps features). 

Intel-BMW i3 integration

Google announced (with Audi, GM, Honda, Hyundai and NVidia) Android in car platform via a global alliance called Open Automotive Alliance (OAA). Get ready for Android homescreens in car (and new levels of tracking :-))

Samsung/BMW announced that Samsung Gear watch can control the car features now.

AR/VR

Chipset companies like Qc, Intel are pushing several AR/VR experiences like gaming & for kids education (e.g. QC & Sesame street partnership). For gaming, Intel has partnered with OEMs for kinect-like RealSense interactions on laptops etc. Played with it a bit, was pretty good at detecting palm/finger-level with a decently accurate depth sensing.

Some Thoughts on privacy

Privacy is soon slated to get out of control for consumers. So far they think they can turn off their one device (or features), but in future there will be smartwatch-like wearables and your car still up for tracking data about you. The expansion path of Smartphones/Tablets ->TVs -> Automotive (Cars) seems to be natural evolution for chipset, OEMs, OS and data mining companies. However, Combine that sort of tracking with data mining of content consumption habits (inputs to your brain) and social network comments (outputs of your brain), and you can almost roughly digitally clone someone’s brain.

Will consumer awareness itself get to a point for them to demand more control of their data, with open technologies like Firefox OS for all their devices? Or would it still be the OEM/carrier partners who want to break free of the OS-maker stranglehold on services and revenues?