Tag Archives: big data

Complete Listing of My Enterprise 2.0 Innovate 2012 Notes

I was pleased to attend Enterprise 2.0 Innovate on the West Coast for the first time. It occurred November 12 – 15 in the Santa Clara Convention Center. Here my notes from this year’s Enterprise 2.0 2012 conference in Boston. Here is a complete listing of my notes from the event.

Enterprise 2.0 Innovate 2012 Notes: The Right Way to Select Emerging Technologies for the Enterprise – Part One

Enterprise 2.0 Innovate 2012 Notes: The Right Way to Select Emerging Technologies for the Enterprise – Part Two

Enterprise 2.0 Innovate 2012 Notes: Living the Hybrid Enterprise

Enterprise 2.0 Innovate 2012 Notes: Tuesday Keynote

Enterprise 2.0 Innovate 2012 Notes: From Project Portfolio to Innovation Funnel

Enterprise 2.0 Innovate 2012 Notes: Wednesday Keynote

Enterprise 2.0 Innovate Notes: 2012 Preventing IT Sprawl

Enterprise 2.0 Innovate 2012 Notes: Tools and Techniques for Visualizing Big Data

Enterprise 2.0 Innovate 2012 Notes: Beyond Adoption

Enterprise 2.0 Innovate Notes: Big Data: Everyone’s Challenge

Enterprise 2.0 Innovate Notes: Open Innovation

Enterprise 2.0 Innovate Notes: Bringing SaaS Innovation to the Enterprise

Enterprise 2.0 Innovate Notes: Beyond Team Member Engagement

Here are the E 2 Innovate Notes from my Merced Group Partner, Catherine Shinners, who covered additional sessions.

E2Innovate: Innovation Inside and Out

E2Innovate: A Collaborative Enterprise: Thoughtworks Leads the Way

Enterprise 2.0 Innovate: Beyond Adoption to Social Process Transformation

Enterprise 2.0 Innovate 2012 Notes: Big Data: Everyone’s Challenge

I was pleased to attend Enterprise 2.0 Innovate on the West Coast for the first time. It occurred November 12 – 15 in the Santa Clara Convention Center. Here are my notes from this year’s Enterprise 2.0 2012 conference in Boston. Here are my notes from the session: Big Data: Everyone’s Challenge, speakers included Aditi Dhagat, Sr. Director, Adobe Information Management, Kaijun Zhan, IT Group Director and Senior Technologist, Cadence Design Systems, Randy Wagner, Drilling Advisor, Apache Corporation, and Johna Till Johnson, Analyst, Nemertes Research.  Here is the session description.

“One of the biggest challenges with Big Data is the fact that it cuts across multiple domains. Everyone in the organization has a stake, from Sales and Marketing to Operations and IT. And even within IT, Big Data poses unique challenges for folks in infrastructure and Applications. How should companies organize to best ensure the success of their Big Data initiatives? This panel of experts discusses pros and cons of Big Data organizational and operational issues. Attendees will leave with an understanding of the challenges, pitfalls, and best practices of organizing a Big Data initiative.”

Johna started the session as the moderator. She said that most companies are not organized right to handle Big Data. This panel looked at this issue and all are doing Big Data work. Kaijun said that there is an IT manager responsible for Big Data and they hired some data scientists from a top university. Aditi said their Big Data effort is in its early mode. It is being driven by their company strategy. They are involved with digital marketing and this is one driver. Another is their suite of creative products. They are moving from a licensing model to a subscription company. They need to see how people are using their products. They want to move people from being free subscribers to paid subscribers.

Randy said his company is involved in oil drilling. They have been dealing with Big Data for along time using seismic data for oil exploration. He has started introducing social tools to the company. They are also looking new uses of Big Data for internal uses. Randy explained that seismic techniques are used to better find oil. They set off explosions and see how sound travels into the ground to understand what is there.

Johna asked about their initial thoughts as they put together their Big Data team. Kaijun said they were trying to do two things: promote business opportunities and increase effectiveness of the organization. One thing they looked at was help desk tickets to see what data was there that could be useful.  They realized they needed to integrate the data scientists so these scientists understood their business model.

Randy asked about how many in the room work in IT and this was about 50% of the room. He said he is non-IT. In his case the business unit is leading the effort and went to the IT people to get it started. The business has been using a form of Big Data for years. Now they need real time data in drilling operations. Currently they spend about 20% of time in non-productive tasks in their drilling. Reducing this is huge cost reduction opportunity. Their long-term goal is to automate drilling operations. IT is a partner but not leading the charge.

Aditi said that Adobe is over 30 years old but still has a start up mentality. They are a software company and have been using Big Data for a long time. Their challenge is a bit different because so many parts of the organization are collecting data in different ways. This has led to some fragmented experiences for their customers. They need to connect all their sources of data. As a proof of concept they did this with aggregated HR data that included content from LinkedIn to predict who will be the future leaders of the company. This impressed senior leadership.  There are also looking at what customers are saying about Adobe. They want to have a better understanding of their customers to micro-target potential customers.

It was asked how you determine what to data mine before you decide how to do it. Aditi said you need to determine the use case. For example, as Adobe changes to a subscription model they want to see how quickly they can move users from the free services to the paid services. Randy said it is not easy to determine what to data mine, as they do not know the unknowns. They are exploring their archive of data to see what is there before they focus.

Kaijun agreed that there is a learning curve in operation. He said that the available tools are expanding. It requires business skills and not just IT skills to properly do this work. Aditi said they are using four categories of tools: capture tools, tools to classify data, then data mining, and finally delivering and targeting data to business processes.  There are also using visualization tools. Their own products come to play here.

Johna asked about how you identify data scientist. Aditi said they look for people interested in statistics, as well as people who do not know their business model so they have a fresh perspective. Kaijun agreed and said they look for people with a creative mind. Randy said that as a company they want to be contrarian and they look for people with this perspective.

It was asked about the need to speed. Randy that for them real time is a few seconds, not nanoseconds. Their data comes from the field through wireless so seconds is possible and they need to respond in minutes so this is okay. Johna asked about data integration of the different types of data. Aditi said that you need a rules engine that can look at the real time data and act as a filter. Randy said they are doing both old school and new school analysis on the same data.

Johna asked about the top three takeaways on Big Data. Randy said there will be many failures so anticipate this. Have your eyes wide open. Make sure you are addressing the right question. Do not skip steps. Aditi said think abut how you are going to connect all the different types of data in a common model. There will be a bit of re-skilling people to build data models. Find a good business unit to partner with. In her case, it is product marketing. Kaijun said that you need to produce fast as people will lose patience.

Enterprise 2.0 Innovate 2012 Notes: Tools and Techniques for Visualizing Big Data

I was pleased to attend Enterprise 2.0 Innovate on the West Coast for the first time. It is occurring November 12 – 15 in the Santa Clara Convention Center. Here my notes from this year’s Enterprise 2.0 2012 conference in Boston. Here are my notes from the session: Can You See Me Now? Tools and Techniques for Visualizing Big Data led by Puneet Piplani, Geography Head, Mu Sigma. Here is the workshop description.

Big Data is creating big headaches, forcing enterprises to find new ways to harness data to make decisions. Big Data is producing a positive wave of disruption in social data analysis and having a major impact on businesses. Third-parties are developing analytical methods and visualization techniques to help companies manage and interpret all data types.  Some may be surprised to learn that there’s a shortage of data scientists and the importance of recruiting analytical talent specifically in the Big Data space is critical.”

Puneet said that Mu Sigma is a decision science and analytics company that was started in 2004. They have over 2,000 scientists working across 10 verticals. He said that data science is simply a means to the end of making better decisions. Data is growing rapidly. Google processes 24 terabytes of data per day. In 1993 the total internet traffic was about 100 terabytes.

The challenge is not just the volume, but also the variety: people to people, people to machines, and machines to machines.  There is also the velocity. There are 2.9 million emails sent every second and 20 hours of video uploaded to YouTube every minute. There are also 50 million tweets per day.

To deal with this exploding data volume you need new applications. The new technology results in more data and all this data is getting stored. Is big data just a hype?

Data is an asset. It can be structured and unstructured. It can be inside and outside the firewall. How do you combine all this data to get useful information?  For example, a sports clothing company wanted to forecast how many team jerseys to create and what players. So they looked at social media to make projections.

A competitive differentiator is how you use the data to make decisions. The consumption of analytics is the differentiator, but the creation. Big data technologies are becoming commoditized bring down the cost of these tools. It is the organizations that learn from the data that will win.

Visualization plays a key role. A Danish scientist looked at how we use our senses. You process in this sense data in the following order of increasing speed: taste, smell, touch, sight. However, the brain process this data slower that it gets all this data. So the brain makes assumptions. The unconscious part of the brain depends more on senses that language. Sight is the fastest sense so visualization is key. He compared a list of figures to a graph. You can see much more from the graph much faster and make assumptions much quicker. The graph frees the brain up from processing numbers to seeing trends.

I was a cognitive psychologist in a former life and saw this power of visualization. Although, there are also times when language trumps visuals. You need to match the characteristics of the media with the cognitive processes that people use to solve problems. Some tasks are best solved through the precision and dichotomies that language offers. Other tasks are best solved with the big picture that visualization offers.

Tools can now provide dynamic visualizations when the data changes and when you ask questions. There can also be personal or role based visualizations that adjust depending on who you are. You can get visual alerts and notifications base don changing data. You need to see the outliers and exceptions. That is where the key data occurs for decision making.

He showed some interesting examples. First, there was a hierarchical tree map that is useful for topic visualizations. One example was foodmood.com.in that showed what people are eating in different places and how they like it and make comparisons (see below).

He then showed thematic rivers to see data trends over time. He went to a site that showed the tweets over time about this conference. Another example is the History of Everything – ChronoZoom (see below).

A third is CNN Ecosphere #RIO20 about a conference in Rio. It is big set of networked dots that you zoom in and explore (see below).  These are public examples and I checked out each Web site. They are fun.

Puneet also mentioned an example his firm created for an airline to minimize the effects of flight delays on over flights. There are many vendors in this space. He showed a 2006 Gartner hype cycle and the visualization of big data was going into the trough of disillusionment. Now it is reaching the slope of enlightenment. It is starting to take hold and be accepted as real and valuable.

This is an area that I am interested in as I am involved with a firm that does data visualization, Darwin Ecosystems. It produces the Darwin Awareness Engine(TM) that looks at what is happening in social media using Chaos Theory based algorithms to let the content self-organize and produces visualizations of the findings. There is also Tweather that focuses specifically on Twitter and shows topic trends over time.

Enterprise 2.0 Innovate 2012 Notes: Wednesday Keynote

I am pleased to attend Enterprise 2.0 Innovate on the West Coast for the first time. It is occurring November 12 – 15 in the Santa Clara Convention Center (see below). Here my notes from this year’s Enterprise 2.0 2012 conference in Boston. Here are my notes from some of the Wednesday Keynote sessions.

Big Data: The Next Industrial Revolution was presented by Rich Carpenter, Chief Technology Strategist, GE Intelligent Platforms that is their industrial software platform. Paige mentioned that GE is the 13th largest software firm in the world.  Rich said that big data is easy to grasp in terms of its scope. But what does this mean for GE? GE operates at the consumer scale through supply chains in such areas as energy. It also supplies many airline engines that they need to keep operational. They are standing at the doorstep of a massive transition in industrial businesses. Now there are more machines connected to the Web than people. This transition is bringing on the industrial internet which he called Web 3.0.

Now applications have built in intelligence and provide analytics that need to be used. There are three main forces in the industrial internet: The Web, smart machines, and big data & analytics. The latter is required to survive in the world of big data generated by the smart machines. One machine can generate 4 trillion samples per year.

GE has over 200,000 connected assets that GE has to run analytics on. It has to make sure its 21,000 plus jet engines are operational, for example. There are also gas and steam turbines and other equipment. GE has large centers to look at trends within its equipment. Their objective is to prevent bad things from happening and the machines operating. This is not an easy task. They have to employ many data scientists to deal with all the data.  The predictive analytics also require deep domain knowledge to identify early warning signs of possible failure.

It is not just a big data challenge. Data diversity is equally challenging. For example, the service of a jet engine and the airplane that uses it are separate tasks. Engines are taken off planes for service and replaced by another engine to keep the plane flying. Then the serviced engine will go to a new plane, perhaps owned by a different airline. All this has to be tracked. You also be able to move from run my machine better to run my fleet better to run my business better.

There are a series of phases in the industrial internet: there is a movement from connect & monitor to intelligence & analytics to business optimization. First you have to solve the data storage problem but that is just the beginning. Then you have to understand it at scale. Finally, when you have the analytic infrastructure in place you need to provide decision support.  There is also a level of accumulated business experience to develop operational models. Their goal is keeping their machines operational and reducing surprises and energy requirements.

Tomorrow’s Challenges for Today’s CIO was a panel moderated by Eric Lundquist, VP & Editorial Analyst for InformationWeek Business Technology Network. It included: Joe Cardenas, CIO, Pacific Compensation Insurance; Marina Levinson, Founder and CEO, CIO Advisory Group, LLC; Shelton Waggner, SVP, Internet2, and Michael Skaff, CIO, LesConcierges.  Eric provided the panel with some questions. CIOs are in the middle of change. They can become vital to the enterprise or become diminished in their importance. How can you make innovation happen as a CIO? CIOs are also moving to becoming bundlers of services rather than builders.

Eric had four questions for the panel – How do you define enterprise innovation? What are the main obstacles to innovation? What is the CIOs role in innovation? What was the most innovative project you were involved with?

Joe said that innovation is about opening people eyes. You need to ask the “what if” questions. You need to discuss more business strategy. The obstacles to innovation include the complexity of the environment and the internal politics in an organization. There is a lot of negotiation involved. The role of the CIO is to bring new ideas to the table and demonstrate the possible. His top project was the creation of a business dialog with multiple partners including no touch transactions that save the customer time and money. It took a lot of negotiation to gain acceptance.

Marina said enterprise innovation is making changes in product offerings, services, business models, and operations that improve the experience of stakeholders. The obstacles are fear of failure, lack of appropriate incentives that do not encourage innovation.  Running the business is a perquisite to having conversations about innovation with business units. You need to be smart where you experiment and not bring the business down. The role of the CIO is now chief innovation officer to explore, partner and define. CIOs need to figure how to add value in this new world. Her top projects were the mobile enterprise portal in 2003 at Palm. It was an IT success even though the company did not succeed.

Shelton said that innovation is supporting individual initiative of every company employee to better support customers. The obstacles are policies that limit, not enable, innovation. Also, not deploying enterprise wide interoperability frameworks. The role of the CIO is to create conditions through open technology to enable individuals to tune enterprise offerings to specific needs. His top project was federated identity to replace in-house platform to use cloud capabilities but maintain corporate coordination. You need to make each employee successful and not just the leaders.

Michael said that innovation is anything that drives the business forward. The obstacles include semantics, communication, and leadership. The role of the CIO is normally one part innovation wellspring, one part facilitator, and ultimately the catalyst in the enterprise. His top project was with the San Francisco Symphony where they projected tweets on the face of City Hall during their Black and White Ball a few years ago. This was innovative at the time as it engaged the audience in a new way.  People discovered this virally. It increased engagement and there was even a marriage proposal that was fortunately accepted.

Unlocking the API Economy was a panel moderated by John Musser, Founder & CEO, ProgrammableWeb. It included: Sam Ramji, VP of Strategy, Apigee; Daniel Jacobson, Director of Engineering – API, Netflex; and Javaun Moradi, Product Manager, NPR. John set the stage as ProgammableWeb has cataloged the development of APIs from about 30 a few years ago to over 8,000. There are many more out there.  Now there are APIs that have over a billion calls a day.

The Netflex’s guy said they have APIs that have several billion transactions a day. NPR has APIs that address over a billion stories a month. Sam said that APIs are emerging as there is a shift from the Web internet to the app internet. Daniel said the multiple devices also support the growing need for APIs. Javaun said that APIs allow you to not say no to things. APIs make you future proof. Supporting the iPad in 5 weeks in one example for NPR. They already had the hard tech stuff done so they could focus on the interface.

The most important strategic decision for API planning is to decide what market you will address. What are your business objectives with whom? This drives your design and execution decisions. At Netflex, their strategy shifted and this affected their API strategy. The NPR guy said to start with user problem that you want to solve. Start small. It is easier to add than subtract capabilities. If even one person is using a feature, it is hard to retire it.

You need to measure intelligently. Tie your APIs to your KPIs. What opportunities that you could address because of the API? Speed is a big feature for APIs so this is one thing to measure. For example, how can you on-board new partners more rapidly?

There are private and public APIs. The public APIs get the most attention. John asked is there a difference between the two. It is better to start with private APIs so you can see the issues.  With public APIs you do not know all the users. With private APIs you can better support the users, as you know them.  You need to see value in the API. If not, no one else will.

If you have a mobile strategy without an API strategy you will not be in business long. Many other devices will come along and APIs can enable you to deal with them.  There are now APIs exchanges in several industries. There are now markets around data exchanges. APIs will only grow as use cases expand.